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General Guidelines Overview 


Welcome to the Search Quality Rating Program! 

As a Search Quality Rater, you will work on many different types of rating projects. The General Guidelines primarily 
cover Page Quality (PQ) rating and Needs Met (NM) rating; however, the concepts are also important for many other 
types of rating tasks. 


For brevity, we refer to “Search Quality Raters” as “raters” in these guidelines. 


~ Should | buy an iPhone? 


Google 


Google Search I'm Feeling Lucky 


Is this page trustworthy? 


Buy cheap iPhones for cheap - BuyCheapiphones.123 


www.buycheapphones.123/buy-cheap-cheaper 
Browse and buy iPhones on the web's number #1 cheap iPhone marketplace. 


Register for FREE! Get cheap iPhones here. 
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Introduction to Search Quality Rating 


0.0 The Search Experience 


The World Wide Web is a vast collection of online information and content. Internet search engines provide a powerful 
way to explore this online universe. There are many ways people search: people may type words into a search box ina 
browser, speak to a mobile phone or assistant device, use search engine autocomplete features, etc. 


People search the Internet for a variety of purposes, ranging from accomplishing a quick task to researching a topic in 
depth. A search may be part of a long-term project, such as a home remodel or vacation planning. A search may be 
done when someone is bored and looking for entertainment, such as a search for [funny videos]. A search may be a 
single question asked during a critical moment of a person's life, such as [what are the symptoms of a heart attack?]. 


Search engines exist to help people find what they are looking for. To do that, search engines must provide a diverse set 
of helpful, high quality search results, presented in the most helpful order. 


Different types of searches need very different types of search results. Medical search results should be high quality, 
authoritative, and trustworthy. Search results for [cute baby animal pictures] should be adorable. Search results for a 
specific website or webpage should have that desired result at the top. Searches that have many possible meanings or 
involve many perspectives need a diverse set of results that reflect the natural diversity of meanings and points of view. 


People all over the world use search engines; therefore, diversity in search results is essential to satisfy the diversity of 
people who use search. 


Finally, search results should help people. Search results should provide authoritative and trustworthy information, not 
lead people astray with misleading content. Search results should allow people to find what they're looking for, not 
surprise people with unpleasant, upsetting, offensive, or disturbing content. Harmful, hateful, violent, or sexually explicit 
search results are only appropriate if the person phrased their search in a way that makes it clear that they are looking for 
this type of content, and there is no other reasonable interpretation of the words used in their search. 


0.1 The Purpose of Search Quality Rating 


As a Search Quality Rater, you will help evaluate search engine quality around the world. Good search engines give 
results that are helpful for people in their own specific language and locale. 


No single rating can directly impact how a particular webpage, website, or result appears in Google Search, nor can it 
cause specific webpages, websites, or results to move up or down on the search results page. Using ratings to position 
results on the search results page would not be feasible, as humans could never individually rate each page on the open 
web. 


Instead, ratings are used to measure how effectively search engines are working to deliver helpful content to people 
around the world. Ratings are also used to improve search engines by providing examples of helpful and unhelpful 
results for different searches. 


As part of your role in the search quality rating program, it is important that you are familiar with and comfortable using a 


search engine. We encourage you to be an expert in search! For example, experiment with using operators (e.g., quotes 
or a hyphen) in your searches or try using advanced search options. 
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0.2 Raters Must Represent People in their Rating Locale 


It is very important for you to represent people in the locale you evaluate. You must be very familiar with the task language 
and location in order to represent the experience of people in your locale. If you do not have the knowledge to do this, 
please inform your employer/company. 


Unless your rating task indicates otherwise, your ratings should be based on the instructions and examples given in these 
guidelines. Ratings should not be based on your personal opinions, preferences, religious beliefs, or political views. 
Always use your best judgment and represent the cultural standards of your rating locale. 


0.3 Browser Requirements 


Check with your employer/company for browser requirements. You may use helpful browser add-ons or extensions, but 
please do not use add-ons or extensions that interfere with or alter the user experience of the page. 


0.4 Ad Blocking Extensions 


Do not use add-ons or extensions that block ads for Needs Met rating or Page Quality rating. These add-ons or 
extensions may cause you to give incorrect ratings. As a rater, only use an ad blocking extension or add-on if specifically 
instructed to do so in the project-specific instructions. 


0.5 Internet Safety Information 


In the course of your work, you will visit many different webpages. Some of them may harm your computer unless you are 
careful. Please do not download any executables, applications, or other potentially dangerous files, or click on any links 
that you are uncomfortable with. 


It is strongly recommended that you have antivirus and antispyware protection on your computer. This software 
must be updated frequently or your computer will not be protected. There are many free and for-purchase 
antivirus and antispyware products available on the web. 


See here for a Wikipedia page on antivirus software and here for a Wikipedia page on spyware. 


We suggest that you only open files with which you are comfortable. The file formats listed below are generally 
considered safe if antivirus software is in place. 


.txt (text files) 

.ppt or .pptx (Microsoft PowerPoint) 
.doc or .docx (Microsoft Word) 

.xls or .xlsx (Microsoft Excel) 

.pdf (PDF files) (Adobe Acrobat) 


If you encounter a page with a warning message, such as “Warning-visiting this web site may harm your computer,” or if 
your antivirus software warns you about a page, you should not try to visit the page to assign a rating. 
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0.6 The Role of Examples in these Guidelines 


The rating concepts in these guidelines apply to all types of content. The examples throughout these guidelines are very 
important to illustrate how the concepts defined in each section should be applied in rating tasks for different types of 
webpages and different types of content. 


Please view each example and keep the following in mind: 


e Webpages and websites change rapidly, so we use images or "snapshots" of webpages in most of our examples. 

e Because some types of content are difficult to view in "snapshot" form, a description of the webpage is provided 
for each example. 

e The information in the examples was accurate at the time it was added, but content and websites may change 
over time. 

e Some examples show pages on desktop and some show pages on mobile devices. 
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Part 1: Page Quality Rating Guideline 


1.0 Introduction to Page Quality Rating 


A Page Quality (PQ) rating task consists of a URL and a grid to record your observations as you explore the landing page 
and the website associated with the URL. The goal of PQ rating is to evaluate how well the page achieves its purpose. 


Here's what you'll need to be a successful Page Quality rater: 


e Your experience using the web as an ordinary user in your rating locale. 
e In-depth knowledge of these guidelines. 
e And most importantly—practice doing PQ rating tasks! 


2.0 Understanding Webpages and Websites 


Page Quality rating requires an in-depth understanding of websites. We'll start with the basics. Along the way, we'll share 
important information about Page Quality rating, so please read through this section even if you are a website expert! 


2.1 Important Definitions 
Here are some important definitions: 
A search engine is a tool to help people find or interact with content available on the Internet. 


In these guidelines, the word "user" refers to a person trying to find information or accomplish a task on the Internet. 
Users are people from many different backgrounds, whose experiences and needs may differ from your own: people of all 
ages, genders, races, religions, political affiliations, etc. 


A webpage is connected to the World Wide Web and can be viewed or "visited" using a web browser (e.g., Chrome), a 
browser on your phone, or a search app. In the 1990s, webpage content was mostly text and links. Today, webpage 
content includes many forms of media (Such as images, videos, etc.) and functionality (such as online shopping features, 
email, calculator functionality, online games, etc.). 


A URL is a string of letters, numbers, and punctuation that your web browser uses to “find” and display a webpage. Page 
Quality rating doesn't require you to have in-depth understanding of the structure of URLs, i.e., you don’t need to know the 
difference between host, domain, etc. But if you are interested, see here to read more. 


A website or site is a group of World Wide Web pages usually containing hyperlinks to each other and made available 
online by an individual, company, educational institution, government, or organization. Popular websites include 
Facebook, Wikipedia, Yahoo, YouTube, etc. 


Note: In these guidelines, we will use the word “website” to refer to a collection of pages owned and controlled by 
a single entity (individual, business, etc.). But we will also use “website” to refer to major “independent” sections 
(or hosts) of some websites that were created to achieve separate purposes. For example, the Yahoo website is 
organized into different sections (or hosts), such as Yahoo Finance (finance.yahoo.com), Yahoo Mail 
(mail.yahoo.com), Yahoo Sports (sports.yahoo.com), etc. Each of these has its own purpose. It’s OK to refer to 
each of these sections as a website; for example, the Yahoo Finance website and the Yahoo Sports website. You 
may also refer to pages on Yahoo Finance or Yahoo Sports as belonging to the Yahoo website. 


A homepage of a website is the main page of the site. It is usually the first page that users see when the site loads. For 
example, http://www.apple.com is the homepage of the Apple site, http:/Awww.yahoo.com is the homepage of the Yahoo 
company site, and hitp://finance.yahoo.com is the homepage of Yahoo Finance. You can usually find the homepage of a 
website by clicking on a “home” link or logo link on subpages of a website. 


A website owner is the person, company, or organization who is responsible for a website. 
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A content creator is the individual(s) or entity (business, organization, etc.) who created the content on a webpage. 


Important: You must be very comfortable exploring websites, both by clicking links and modifying URLs in the address 
bar of your web browser. Become a website detective and explorer! 


2.2 Understanding the Purpose of a Webpage 


The purpose of a page is the reason or reasons why the page was created. Every page on the Internet is created for a 
purpose, or for multiple purposes. Most pages are created to be helpful for people, thus having a beneficial purpose. 
Some pages are created merely to make money, with little or no effort to help people. Some pages are even created to 
harm users. The first step in understanding a page is figuring out its purpose. 


Why is it important to determine the purpose of the page for PQ rating? 


e The goal of PQ rating is to determine how well a page achieves its purpose. In order to assign a rating, you must 
understand the purpose of the page and sometimes the website. 

e By understanding the purpose of the page, you'll better understand what criteria are important to consider when 
evaluating that particular page. 

e Websites and pages should be created to help people. If that is not the case, a rating of Lowest may be 
warranted. More on this later. 


As long as the page is created to help people, we will not consider any particular page purpose or type to be higher quality 
than another. For example, encyclopedia pages are not necessarily higher quality than humor pages. 


Important: There are highest quality and lowest quality webpages of all different types and purposes: shopping pages, 
news pages, forum pages, video pages, pages with error messages, PDFs, images, gossip pages, humor pages, 
homepages, and all other types of pages. The type of page does not determine the PQ rating—you have to understand 
the purpose of the page to determine the rating. 


Common helpful or beneficial page purposes include (but are not limited to): 


To share information about a topic. 

To share a personal experience, perspective, or feelings on a topic. 
To share pictures, videos, or other forms of media. 

To demonstrate a personal talent or skill. 

To express an opinion or point of view. 

To entertain. 

To offer products or services. 

To allow users to post questions for other users to answer. 

To allow users to share files or to download software. 


Here are a few examples where it is easy to understand the purpose of the page: 


Type of Page Purpose of the Page 

News website homepage To inform users about recent or important events. 
Shopping page To sell or give information about the product. 

Video page To share a cute video of a cat. 

Currency converter page To calculate equivalent amounts in different currencies. 


Here is an example (OmNomNomNom Page) of a helpful page where the purpose of the page is not as obvious. At first 
glance, this page may seem pointless or strange. However, it is a page from a humorous site that encourages users to 


post photos with mouths drawn on them. The purpose of the page is humor or artistic expression. This page has a 
helpful or beneficial purpose. Even though the About page on this website is not very helpful, the website explains itself 
on its FAQ page. 
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2.3 Your Money or Your Life (YMYL) Topics 


Pages on the World Wide Web are about a vast variety of topics. Some topics have a high risk of harm because content 
about these topics could significantly impact the health, financial stability, or safety of people, or the welfare or well-being 
of society. We call these topics “Your Money or Your Life” or YMYL. 


YMYL topics may significantly impact or harm one or more of the following: 
e the person who is directly viewing or using the content 
e other people who are affected by the person who viewed the content 
e groups of people or society affected by the actions of people who viewed the content 


YMYL topics can directly and significantly impact people’s health, financial stability or safety, or the welfare or well-being 
of society, because of the following reasons: 

e The topic itself is harmful or dangerous. For example, there is clear and present harm directly associated with 
topics related to self-harm, criminal acts, or violent extremism. 

e The topic could cause harm if the content is not accurate and trustworthy. For example, mild inaccuracies 
or content from less reliable sources could significantly impact someone's health, financial stability, or safety, or 
impact society, for topics like: symptoms of a heart attack, how to invest money, what to do if there is an 
earthquake, who can vote, or needed qualifications for obtaining a driver’s license. 


To determine whether a topic is YMYL, assess the following types of harm that might occur: 
e YMYL Health or Safety: Topics that could harm mental, physical, and emotional health, or any form of safety 
such as physical safety or safety online. 
e YMYL Financial Security: Topics that could damage a person's ability to support themselves and their families. 
e YMYL Society: Topics that could negatively impact groups of people, issues of public interest, trust in public 
institutions, etc. 
e YMYL Other: Topics that could hurt people or negatively impact welfare or well-being of society. 


It's possible to imagine a hypothetical harmful page for any non-harmful topic, such as the science behind rainbows or 
shopping for pencils: for either of these topics, someone could build a page that has a malicious computer virus download. 
However, for a specific topic to be YMYL, the topic itself must potentially impact people’s health, financial stability, or 
safety, or the welfare or well-being of society. 


Many or most topics are not YMYL and do not require a high level of accuracy or trust to prevent harm. Because YMYL 
assessment is a spectrum, it may be helpful to think of topics as clear YMYL, definitely not YMYL or something in 
between. Pages on clear YMYL topics require the most scrutiny for Page Quality rating. 
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Type of Topic 


Information 


Could significant harm result 
from inaccurate information? 


Advice about an activity 


Could significant harm result 
from poor advice? 


A personal opinion 


What impact could this opinion 
have on other people and 
society? 


News about current events 


Could this topic significantly 
impact people and society? 


For societal impact, consider 
issues such as elections and 
trust in public institutions that 
benefit society. 


Sharing on social media 


Could the social media post 
cause significant harm? 
Could it hurt individuals? 
Could it damage society if 
widely shared? 


Online commerce and 
product reviews 


Consider the product. Could 
the product cause significant 
harm? 


Clear YMYL Topic 


Evacuation routes for a 
tsunami 


Explanation: Inaccurate 
information on evacuation 
routes could cause significant 
harm to people. 


When to go to the 
emergency room 


Explanation: Bad advice on 

when to go to the emergency 
room could cause significant 
harm. 


Personal opinion about why 
a racial group is inferior 


Explanation: Pages on this 
topic have been used to justify 
or incite violence against 
groups of people. 


News about ongoing 
violence 


Explanation: People need 
accurate information to stay 
safe. Society may also be 
impacted by information about 
ongoing violence, as citizens 
and governments make civic 
decisions accordingly. 


A tide pod challenge post 


Explanation: This harmful 
social media challenge was 
responsible for deaths. 


Purchasing prescription 
drugs 


Explanation: Prescription 
drugs have the potential to 
cause harm and require 
purchase from licensed 
pharmacies. 


May be YMYL Topic 


Weather forecast 


Explanation: In most 
situations, slightly inaccurate 
information about the weather 
forecast will not cause harm. 
People often ask family 
members "what's the weather 
today". 


How often to replace a 
toothbrush 


Explanation: This is a casual 
health topic people commonly 
discuss with friends. A slightly 
imperfect suggestion is 
unlikely to significantly impact 
health or safety. 


Personal opinion about why 
an exercise is inferior 


Explanation: While there may 
be a health concern if the 
exercise is extreme or risky, 
most discussions of jogging vs 
swimming, etc. involve 
personal preference. 


News about a car accident 


Explanation: The accident 
itself may have been harmful, 
but there is likely little risk of 
future harm from small 
inaccuracies in reporting 
about an incident. 


A hot sauce challenge 


Explanation: While some 
people may experience some 
discomfort by tasting various 
hot sauces, it is unlikely that 
sharing about such challenges 
would cause significant harm. 


Review of a type of car 


Explanation: While cars are 
big purchases, many people 
ask friends and family about 
Cars. 
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Music award winners 


Explanation: This topic is 
unlikely to cause harm. 


How frequently to wash 
jeans 


Explanation: This topic is 
unlikely to cause harm. 


Personal opinion about why 
a rock band is inferior 


Explanation: This topic is 
unlikely to cause harm, 
although there may be strong 
opinions involved! 


News about a local high 
school basketball game 


Explanation: This topic is 
unlikely to cause harm. 


A music video 


Explanation: This type of 
content generally has little risk 
of harm. 


Purchasing pencils 


Explanation: Pencils and 
other everyday items are 
unlikely to cause harm. 
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If you are having trouble deciding whether a topic is YMYL, consider the following questions: 
1. Would a careful person seek out experts or highly trusted sources to prevent harm? Could even minor 
inaccuracies cause harm? If yes, then the topic is likely YMYL. 
2. Is the specific topic one that most people would be content with only casually consulting their friends about? If 
yes, the topic is likely not YMYL. 


Important: For pages about clear YMYL topics, we have very high Page Quality rating standards because low quality 
pages on such topics could potentially negatively impact a person’s health, financial stability, or safety, or the welfare or 
well-being of society. 


2.4 Understanding Webpage Content 


All of the content on a webpage can be classified as one of the following: Main Content (MC), Supplementary Content 
(SC), or Advertisements/Monetization (Ads). In order to understand the purpose of a webpage and do PQ rating, you will 
need to be able to distinguish among these different parts of the page. 


Webpage design can be complicated, so make sure to click around and explore the page. See what kind of content is 
behind the tabs and test out the interactive page features. Content behind the tabs may be considered part of the MC, 
SC, or Ads, depending on what the content is. 


2.4.1 Identifying the Main Content (MC) 


Main Content is any part of the page that directly helps the page achieve its purpose. MC can be text, images, videos, 
page features (e.g., calculators, games), and it can be content created by website users, such as videos, reviews, articles, 
comments posted by users, etc. Tabs on some pages lead to even more information (e.g., customer reviews) and can 
sometimes be considered part of the MC. 


The MC also includes the title at the top of the page (example). Descriptive MC titles allow users to make informed 
decisions about what pages to visit. Helpful titles summarize the MC on the page. 


Type of Page and Purpose MC Highlighted in Yellow 
News website homepage: the purpose is to inform users about recent or important events. MC - News Homepage 
News article page: the purpose is to communicate information about an event or news topic. MC - News Article 


Store product page: the purpose is to sell or give information about the product. 
e Content behind the Reviews, Shipping, and Safety Information tabs are considered to be |MC - Shopping Page 
part of the MC. 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies. MC - Currency Converter 

Blog post page: the purpose is to share music used on a TV show. MC - Blog Post Page 

Search engine homepage: the purpose is to allow users to enter a query and search the Internet. |MC - Search Engine Homepage 
Bank login page: the purpose is to allow users to log in to bank online. MC - Bank Login Page 
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2.4.2 Identifying the Supplementary Content (SC) 

Supplementary Content contributes to a good user experience on the page, but does not directly help the page achieve its 
purpose. SC is an important part of the user experience. One common type of SC is navigation links that allow users to 
visit other parts of the website. In some cases, content behind tabs may be considered part of the SC. 


Sometimes the easiest way to identify SC is to look for the parts of the page that are not MC or Ads. 


Type of Page and Purpose SC Highlighted in Blue 
News article page: the purpose is to communicate information about an event or news topic. SC - News Article 

Store product page: the purpose is to sell or give information about the product. SC - Shopping Page 

Blog post page: the purpose is to share music used on a TV show. SC - Blog Post Page 


2.4.3 Identifying Advertisements/Monetization (Ads) 


Ads may contribute to a good user experience. Advertisements/Monetization (Ads) is content and/or links that are 
displayed for the purpose of monetizing (making money from) the page. The presence or absence of Ads is not by itself a 
reason for a High or Low quality rating. Without advertising and monetization, some webpages could not exist because it 
costs money to maintain a website and create high quality content. 


There are many different ways to monetize a webpage, including advertisements and affiliate programs. See here for 
more information on website monetization. Note that monetization on mobile pages may be more subtle than 
monetization on desktop pages. 


The most common type of monetization is advertisements. Ads may be labeled as "ads," "sponsored links," “sponsored 
listings,” “sponsored results,” etc. Ads may change when you reload the page, and different users may see different Ads 
on the same page. 


Website owners can choose to display Ads on their page (such as by joining an advertising network), but they may not 
always directly control the content of the Ads. However, we will consider a website responsible for the overall quality of 
the Ads displayed. 


Important: For the purpose of these guidelines, we will consider monetized links of any type to be “Ads.” See here for 
different types of website monetization. 


Type of Page and Purpose Ads Highlighted in Red 
News article page: the purpose is to communicate information about an event or news topic. Ads - News Article 

Blog post page: the purpose is to share music used on a TV show. Ads - Blog Post Page 
Store product page: the purpose is to sell or give information about the product. No ads — Shopping Page 


2.4.4 Summary of the Parts of the Page 
Let's put it all together. 


e Main Content (MC) is any part of the page that directly helps the page achieve its purpose. MC is (or should be!) the 
reason the page exists. The quality of the MC plays a very large role in the Page Quality rating of a webpage. 

e Supplementary Content (SC) is also important. SC can help a page better achieve its purpose or it can detract from 
the overall experience. 

e Many pages have advertisements/monetization (Ads). Without advertising and monetization, some webpages 
could not exist because it costs money to maintain a website and create high quality content. The presence or 
absence of Ads is not by itself a reason for a High or Low quality rating. 


On some pages, user reviews and comments may be considered MC, and on other pages they may be considered SC. 
Use your best judgment and think about the purpose of the page. 
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Do not worry too much about identifying every little part of the page. Think about which parts of the page are the MC. 
Next, look for the Ads. Anything left over can be considered SC. 


Type of Page and Purpose MC, SC, & Ads Highlighted 
News article page: the purpose is to communicate information about an event or news topic. Summary - News Article 

Store product page: the purpose is to sell or give information about the product. Summary - Shopping Page 
Currency converter page: the purpose is to calculate equivalent amounts in different currencies. Summary - Currency Converter 
Blog post page: the purpose is to share music used on a TV show. Summary - Blog Post Page 
Bank login page: the purpose is to allow users to log in to the bank online. Summary - Bank Login Page 


2.5 Understanding the Website 


Pages often make more sense when viewed as part of a website. Some of the criteria in Page Quality rating are based 
on understanding the website that the page belongs to. 


To understand a website, start by finding out who is responsible for the website and who created the content on the page 
(more information provided in Section 2.5.2). Then, look for information about the website and/or content creators on the 
website itself. Websites and content creators are usually very eager to tell you all about themselves! 


You must also look for reputation information about the website and/or content creators. What do outside, independent 
sources say about them? When there is disagreement between what the website or content creators say about 
themselves and what reputable independent sources say, trust the independent sources. 


2.5.1 Finding the Homepage 


The homepage of a website usually contains or has links to important information about the website. Website owners 
usually make it easy to get to the homepage from any page on the site. 


Here's how to find the homepage of a website: 


1. Examine the landing page of the URL in your PQ rating task. 
2. Find and click on the link labeled with the name or logo of the website (occasionally labeled as “home" or “main"), 
which usually appears at the top of the page. 


Sometimes, you may be given a webpage or website that appears to have no navigation links, no homepage link, and no 
logo or other means to find the homepage. Even some High or Highest quality pages lack a way to navigate to the 
homepage. If you can't find a link to the homepage, modify the URL by removing everything to the right of “.com,” “.org,” 
“net,” “fr,” etc. and refresh the page. 


Occasionally, your rating task will include a URL for which there are two or more justifiable “homepage” candidates. For 
example, you may not be sure whether the homepage of the URL hittps://finance.yahoo.com/calendar is 


http://finance.yahoo.com or _http://www.yahoo.com. 
Important: When you have more than one homepage “candidate,” please use whichever one offers the most information 


about the specific webpage in the rating task. Use your judgment. The goal is to understand the webpage and the 
website(s) it is associated with, not find the one unique, correct homepage. 


Copyright 2023 15 


In the following examples, we have included the URL of the page to be evaluated in the rating task, as well as the URL of 
its associated homepage. We have also included an image that shows where to click on the landing page to navigate to 

the homepage. In the image, you will see a red box around the link or logo you would click to navigate to the homepage. 
URL of the Task Page Homepage of the Website Where to click to get to the Homepage 


Williams-Sonoma Homepage 


ee ee WILLIAMS 


om/products/shun-premier.7_ | Ltt2:/Avww.williams-sonoma.com SO} OMA 
piece-knife-block-set 


This “WILLIAMS-SONOMA” logo shown in the 
upper center of the page is clickable and takes 
users to the homepage of the website. 


http://hms.harvard.edu Harvard Medical School Facts and Figures Page 

In this case, we will consider the Harvard Medical sae 

School page at http://hms.harvard.edu to be the Eg H A RVA R D 
http://hms.harvard.edu/about- | homepage, rather than http://www.harvard.edu MEDICAL SCHOOL 
hms/facts-figures (which is the homepage of Harvard University). 


Clicking the logo at the top of This “Harvard Medical School” logo in the upper 
http://nms.harvard.edu/about-hms/facts-figures left part of the page is clickable and takes users 


takes users to http://nms.harvard.edu, not to to the homepage of the Harvard Medical School 
http://www. harvard.edu. website. 


2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page 
Every page belongs to a website, and it should be clear: 


e Who (what individuals, company, business, organization, government agency, etc.) is responsible for the website. 

e Who (what individuals, company, business, organization, government agency, etc.) created the content on the 
page you are evaluating. Note that for pages on websites such as forums and social media platforms, people 
may post content using an alias or username in order to avoid sharing personally identifiable information online. 
In these cases, the alias or username is an acceptable way to identify the content creator. 


Websites are usually very clear about who is responsible for the website and who created the content on the page. There 
are many reasons for this: 


People may want to showcase their own personal talent, experiences, and perspectives. 

Artists, authors, musicians, and other original content creators may want to be known and appreciated. 
Commercial websites may have copyrighted material they want to protect. 

Businesses and organizations may want people to know who they are and what they do. 

Stores want customers to feel comfortable buying their products online. 


Most websites have “contact us” or “about us” or “about” pages that provide information about who owns the site. Many 
companies have an entire website or blog devoted to who they are and what they are doing, what jobs are available, etc. 
Here are some examples: 


Google Official Blog 
Marriott Blog 
Southwest Airlines Blog 
Netflix Tech Blog 
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In PQ rating tasks, you will need to identify who created the MC on the page. To help you do so, the table below 
describes the primary content creators for common webpage types. These types can overlap (e.g., a page may have 
multiple articles, some written by individual authors or journalists and others written by the website owner). 


Webpage Type 


The website owner created the page 
and much of the MC on the page 


The page may have comments, reviews, 
or other content posted by users, but the 
page itself is the responsibility of the 
website. 


The website owner created the page, 
and the MC is produced by authors or 
other content creators identified by 
the website 


The website decides what to publish and 
is responsible for the content, but there 
are distinct authors or content creators 
who provide the MC on the page. 


The webpage consists of social media 
post(s) from a single account 
representing an individual content 
creator or organization 


The social media website owner enables 
people and organizations to create 
accounts to post text, images, videos, 
and other types of content under their 
account. 


The webpage is created by multiple 
users engaging in discussion or 
posting on social media 


The website owner enables people to 
post text, images, and videos or have 
conversations with other users of the 
website. 


Examples 


Homepage of a business 
website 

Introductory page ona 
personal website 

Product page on an online 
merchant website 


Newspaper opinion piece 
written by the editorial 
board 

Magazine article written by 
an individual journalist 
Scientific journal paper 
written by a team of 
academic researchers 


Social media post 

Local business profile 
page on a social media 
website 

Video channel on a video 
sharing website 


Forum discussion thread 
Q&A question page 
Search results page ona 
social media website 
showing content from 
many different users 
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Primary Content Creator(s) 


The website itself can be viewed as the 
primary content creator. Website owners 
may have content created on their behalf 
(e.g., a small business may hire a 
professional web developer to build their 
website), but they are ultimately 
responsible for the MC. 


User comments and reviews may play a 
significant role on the page. As long as 
the website creates and actively 
maintains the page, the website is 
considered to be the primary content 
creator. 


The primary content creators are the 
journalists, scientists, etc. listed as 
authors of the content. Often the content 
creators are individuals, but an 
organization, company, or institution may 
also be the content creator. 


User comments may be present, but they 
typically are not the focus of the page. 


The primary content creator is the person 
or organization who created the account 
and is posting the MC. There may be 
information about the content creator on 
a profile page found on the website. 


Other user comments and reactions such 
as "likes" may be considered part of the 
MC. 


The people posting are the content 
creators. 
There is no single primary content 


creator, and people may be identified 
only by aliases or usernames. 
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2.5.3 Finding About Us, Contact Information, and Customer Service Information 


Many websites are interested in communicating with their users. There are many reasons that users might have for 
contacting a website, from reporting problems such as broken pages, to asking for content removal. Many websites offer 
multiple ways for users to contact the website: email addresses, phone numbers, physical addresses, web contact forms, 
etc. Sometimes, this contact information is even organized by department and provides the names of individuals to 
contact. 


L.L.Bean mm 20” OFF ERRE ho 


MEN'S WOMEN'S KIDS FOOTWEAR OUTDOORGEAR HUNTING & FISHING LUGGAGE HOME SALE Lt BEAN 
- SIGNATURE 
) Coupons with the LL Bean? Visa® Card WhatsNew Flannel Gifs GiftCards Biog ParkFinder Videos LLBean Visa 


Ordering Information 
Order Status 

My Account 

Shipping Information 
Returns & Exchanges 
Sizing & Fit 
Monogramming & Product 
Services 

Gift Options 

Wish List QUICK LINKS 
Catalogs » Fon 
Outdoor Discovery Schools 
International Help » Size Charts rR 
Technical information 


100 YEARS of OUTSTANDING CUSTOMER SERVICE” 


een the guiding principle 
ore, during or after you 


hoose a link below 


issword » Order Tracking 


d Exchanges 
CONTACTUS: 4} Phone t call Me ® Live Help 3 Email More Contact info 
Helpful Links 

100% Guaranteed 

Contact Us 

Find a Store 


Ordering Information Shipping Information Monogramming & Outdoor Discovery Schools 
biais Product Services m tion 


LLBean Signature 
Product Recalls 
Your Privacy 

About Security 


& Exchanges Gift Options 
& Exchanges Gift Boxing 
Gift Message 


Catalogs Technical information 
atal Web Browser 


The types and amount of contact information needed depend on the type of website. Contact information and customer 
service information are extremely important for websites that handle money, such as stores, banks, credit card 
companies, etc. Users need a way to ask questions or get help when a problem occurs. 


For shopping websites, we'll ask you to do some special checks. Look for contact information—including the store’s 
policies on payment, exchanges, and returns. Sometimes this information is listed under “customer service.” 


Some kinds of websites need fewer details and a smaller amount of contact information for their purpose. For example, 
humor websites may not need the level of detailed contact information we would expect from online banking websites. 


Occasionally, you may encounter a website or content creator with a legitimate reason for anonymity. For example, 
personal websites may omit personal contact information such as an individual’s home address or phone number. 
Similarly, websites with user-generated content may allow the author to identify themself with an alias or username only. 


To find contact or customer service information for a website, start with the homepage. Look for a “contact us” or 
“customer service” link. Explore the website if you cannot find a “contact us” page. Sometimes you will find the contact 
information on a “corporate site” link or even on the company’s official social media page. Be a detective! 


Note that different locales and different social media platforms may have their own conventions and standards for what 
information should be available on the website. Please use your judgment and knowledge of your locale. 
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3.0 Overall Page Quality Rating 
Now that you are an expert in understanding websites and webpages, here are the high-level steps of Page Quality rating: 


1. Assess the true purpose of the page. If the website or page has a harmful purpose or is designed to deceive 

people about its true purpose, it should be rated Lowest. 

2. Assess the potential of the page to cause harm as described in these guidelines. Websites or pages that are 

harmful to people or society, untrustworthy, or spammy as defined in these guidelines should be rated Lowest. 

3. Otherwise, the PQ rating is based on how well the page achieves its purpose as described in these guidelines. 
Important: There are harmful pages that are seemingly “official”, “expert”, or “authoritative”. Any type of website may 
have pages with harmful MC, from user-uploaded videos and forum posts to harmful products sold online to pages to 
pages that mimic the look of scientific papers or encyclopedia entries. All pages should be evaluated for harm—including 
pages on government websites, academic institutions, reputable online stores, charities, or other types of generally helpful 
websites. 


On Page Quality rating tasks, you will use the Page Quality sliding scale (slider) to assign the overall PQ rating. The 
slider looks like this: 


| | | | | 


N/A Lowest Lowest+ Low Low+ Medium Medium+ High High+ Highest 


You may also use the in-between ratings of Lowest+, Low+, Medium+, and Hight. Please interpret the “+” as “+ %2,” 
meaning that the Lowest+ rating is halfway between Lowest and Low, Low+ is halfway between Low and Medium, etc. 


Identifying Lowest and Low quality pages can be the most difficult part of PQ rating. The guidance in the sections on 
Lowest and Low is more extensive to help you rate accurately. High and Highest quality pages are typically easier to 
identify, but be sure to apply the standards and examine the examples throughout these guidelines to help calibrate your 
ratings. As might be expected, Medium quality pages are commonly found online. 


3.1 Page Quality Rating Considerations 


The following table summarizes the things that you should consider in Page Quality rating. Upcoming sections of these 
guidelines will elaborate on these considerations and explain how they apply to pages across the PQ rating scale. 


Consideration Why it’s Important for Page Quality Rating 

The purpose of the page If the page has a harmful purpose, or if it is designed to deceive people about 
its true purpose or who is responsible for the content, it should be rated 
Lowest. 


Otherwise, PQ rating is the process of determining how well a page achieves 
its purpose. 


The potential for the page or website Websites or pages that are harmful to people or society, untrustworthy, or 
to cause harm as described in these spammy as described in these guidelines should be rated Lowest. 
guidelines 


The topic of the page and the extent The topic of the page helps determine the standards for your overall PQ 


to which that topic is YMYL assessment. Pages on YMYL topics have higher standards than pages on 
non-YMYL topics. 
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Consideration 


The type of website 


Information provided by the website 
and content creator 


Quality of the MC 


The title of the page 


The role of Ads and SC on the page 


Reputation of the website and content 
creator 


Trustworthiness of the page: E-E-A-T 


Why it's Important for Page Quality Rating 


Different types of websites and webpages have different expectations for PQ 
rating. For example, PQ expectations may differ for: 


e Small hobbyist websites vs. large corporate websites 

e Websites involving financial transactions vs. websites that do not 
require payment or collect personal information 

e Websites with content created by ordinary people on a volunteer basis 
vs. websites with content created by professionals 


While the information that websites and content creators provide about 
themselves isn't always trustworthy, it can provide an important starting point. 


Consider the extent to which the MC is satisfying and helps the page achieve 
its purpose. 


The title of the page is considered part of the MC. Descriptive MC titles that 
summarize the page allow people to make informed decisions about what 
pages to visit. 


Consider the ways in which the Ads and SC contribute to how people 
experience the page. 


Remember: Many websites need monetization to share content with users. 
The presence or absence of Ads alone is not a consideration for PQ rating. 


Research the reputation of the website and the content creator to learn about 
how others view the website and who is behind it. 


Assess how trustworthy the page is. Experience, Expertise and 
Authoritativeness can help with your assessment of Trust. Some types of 
pages require a high level of Trust. 


Important: These considerations overlap. For example, while examining the quality of the MC, you may notice factual 
inaccuracies that lower your assessment of Trust. While conducting reputation research, you may find information about 
the expertise of the content creator which increases your level of Trust. This is how PQ rating is designed to work! 
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3.2 Quality of the Main Content 


The quality of the Main Content (MC) is one of the most important considerations for PQ rating. The MC plays a major 
role in determining how well a page achieves its purpose. 


The unifying theme for evaluating the quality of the MC is the extent to which the MC allows the page to achieve its 
purpose and offers a satisfying user experience. For most pages, the quality of the MC can be determined by the amount 
of effort, originality, and talent or skill that went into the creation of the content. For informational pages and pages on 
YMYL topics, accuracy and consistency with well established expert consensus is important. 


Effort: Consider the extent to which a human being actively worked to create satisfying content. Effort may be 
direct, such as a person translating a poem from one language to another. Effort may go into designing page 
functionality or building systems that power a webpage, such as the creation of a page that offers machine 
translation as a service to users. On the other hand, the automatic creation of thousands of pages by running 
existing freely available content through existing translation software without any oversight, manual curation, 
etc., would not be considered to have human effort. 


De 


For pages like social media posts or forum discussions, the level of participation and depth of conversation is 
an important part of effort. Contributions from multiple individuals on such pages can add up to a significant 
amount of total human effort. 


# | Originality: Consider the extent to which the content offers unique, original content that is not available on 


2 other websites. If other websites have similar content, consider whether the page is the original source. 
2; Talent or Skill: Consider the extent to which the content is created with enough talent and skill to provide a 
= 


satisfying experience for people who visit the page. 


Accuracy: For informational pages, consider the extent to which the content is factually accurate. For pages 
on YMYL topics, consider the extent to which the content is accurate and consistent with well-established 
expert consensus. 


The purpose of the page, topic of the page, and type of website all play a role in how to evaluate the quality of the MC. 
For example, consistency with well-established expert consensus is important for medical advice. Skill is important for 
how-to videos. Talent and originality is important for artistic expression. The amount of effort expected for a short video 
shared on social media is less than for a full-length, professionally produced documentary on a streaming video website, 
but both need sufficient effort to create satisfying content for their purpose. Think about what effort, originality, talent, or 
skill looks like for the type of page that you are evaluating. 


For each page you evaluate, spend a few minutes examining the MC before drawing a conclusion about it. Read the 
article, watch the video, examine the pictures, use the calculator, play the online game, etc. Remember that MC also 
includes page features and functionality, so test the page out. For example, if the page is a product page on a store 
website, put at least one product in the cart to make sure the shopping cart is functioning. If the page is an online game, 
try to play the game yourself. Do your best to imagine that you are someone who's very interested in the topic, 
functionality, or purpose served by the page, then think about how satisfying the MC would be for that person. 


High and low quality MC comes in all formats (e.g., text, audio, video, images) and all lengths (e.g., short-form videos and 
full-length professional documentaries). High and low quality content also exists on all types of websites, from small 
personal sites to large corporate sites, from forums and social media to websites that handle financial transactions. Think 
carefully about what helps the page achieve its purpose and what makes the MC satisfying for users. 
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3.3 Reputation of the Website and Content Creators 


An important part of PQ rating is understanding the reputation of the website. If the website is not the primary creator of 
the MC, it’s important to research the reputation of the content creator as well. 


Reputation research should be performed according to the topic of the page. For example, if the page contains medical 
information, research the reputation of the website and content creator for providing medical information. It's possible for 
a website to be a go-to source for one type of content (e.g., humorous videos), but an untrustworthy source for a different 
type of content (e.g., financial information). 


A website's or content creator's reputation can also help you understand what a website or content creator is best known 
for. For example, newspapers may be known for high quality, independent investigative reporting while satire websites 
may be known for their humor. An individual journalist (content creator) may be known for the clarity of their scientific 
articles while a food blogger (content creator) may be known for the deliciousness of the recipes they post online. 


Note that a company or person may create content on many different websites. For example, a newspaper might have 
their own website, upload their video content to a video sharing website, post updates on social media, and contribute 
content to a TV channel. An expert on a topic might publish research papers, have a lengthy blog, and share short 
updates on social media. In these cases, you should research the underlying company or content creator. 


Reputation research is especially important for detecting untrustworthy websites and content creators. Content may look 
great on the surface, but reputation research can expose scams, fraud, or other signs of harm. You never know what you 
will find unless you look! Therefore, reputation research is required for all PQ rating tasks. 


3.3.1 Reputation of the Website 


A website's reputation is based on the experience of real users and the opinions of people who are experts. Websites 
may represent real companies, organizations, and other entities. Reputation research applies to both the website and the 
actual company, organization, or entity that the website is representing. 


Many websites are eager to tell users how great they are. Your job is to independently evaluate the Page Quality of the 
website, not just accept information that appears on one or two pages of the website without further verification. Be 
skeptical of claims that websites make about themselves, particularly when there is a clear conflict of interest. 


Instead, look for independent reviews, references, recommendations by experts, news articles, and other sources of 
credible information about the website. Look for information written by a person or organization, not statistics or other 
machine-compiled information. News articles, Wikipedia articles, blog posts, magazine articles, forum discussions, and 
ratings from independent organizations can all be great sources of reputation information. 


For YMYL topics, the reputation of a website should be judged by what experts in the field have to say. 
Recommendations from expert sources, such as professional societies, are strong evidence of a positive reputation. 


Sources of reputation information will also vary according to the topic or type of company/organization/entity that the 
website represents. For example, you might find that a newspaper (with an associated website) has won journalistic 
awards. Prestigious awards or a history of high-quality original reporting are strong evidence of positive reputation for 
news websites. 


Note that some types of information about a website is not related to its reputation. For example, websites like Similarweb 


have information about Internet traffic to the website, but do not provide evidence of positive or negative reputation. You 
can ignore this type of information since it's not helpful for PQ rating. 
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3.3.2 Customer Reviews as Reputation Information 


Customer reviews can be helpful for assessing the reputation of a store, business, or any website that offers products or 
services to users. You may consider a large number of detailed, trustworthy, positive user reviews as evidence of positive 
reputation for a store or business. 


However, you should interpret customer reviews with care, particularly if there are only a few. Keep the following in mind: 


Be skeptical of both positive and negative reviews. Anyone can write them, including the website owner or 
someone whom the store or business hires for this purpose. 

Try to find as many reviews as possible. Any store or business can get a few negative reviews—this is 
completely normal and expected. Large stores and companies receive thousands of reviews, and most receive 
some negative ones. 

Read the reviews because the content of the reviews matter, not just the number or star rating. Credible, 
convincing reports of fraud and financial wrongdoing is evidence of extremely negative reputation. On the other 
hand, a single encounter with a rude clerk or the delayed receipt of a single package should not be considered 
negative reputation information. Please use your judgment. 


3.3.3 How to Search for Reputation Information about a Website 


Here is how to research the reputation of the website: 


1. 


Identify the “homepage” of the website. For example, for the IBM website, ibm.com is the homepage. You may 
need to identify the content creator, if it is different from that of the overall website. 


Using ibm.com as an example, try one or more of the following searches on Google: 


[ibm -site:ibm.com] - A search for IBM that excludes pages on ibm.com. 

[“ibm.com’” -site:ibm.com] - A search for “ibm.com” that excludes pages on ibm.com. 

[ibm reviews -site:iom.com] - A search for reviews of IBM that excludes pages on ibm.com. 

[“ibm.com” reviews -site:ibm.com] - A search for reviews of “ibm.com” that excludes pages on ibm.com. 


Note: When searching for reputation information, try to find sources that were not written or created by the 
website or the company itself. For example, IBM might have official social media pages that it closely maintains, 
which would not be considered independent sources of reputation information about the company. See here for a 
Wikipedia article on identifying and using independent sources. 


Look for articles, references, recommendations by experts, and other credible information written by people about 
the website. 


High quality news articles and informational articles may be good sources of reputation information. Search for 
such articles. For example, try [ibm site:en.wikipedia.org]. News articles and informational articles can help you 
learn about a company and may include information specific to reputation, such as awards and other forms of 
recognition, or also controversies and issues. Note that some informational articles include a message warning 
users that there are disagreements on some of the content, or that the content may be outdated. This may be an 
indication that additional research is necessary. 


Make sure the information you find is appropriate for judging the reputation of the website. For example, 


reputation information for YMYL topics should come from sources that have expertise in the associated YMYL 
topic. 
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Here are some examples of reputation information: 


annualcreditreport.com 


clevelandclinic.org 


csmonitor.com 


kernel.org 


Site selling children’s 
jungle gym 


Site selling products 
related to eyewear 


Organization serving the 
hospitalized veteran 
community 


Search results for [annualcreditreport.com 
-site:annualcreditreport.com 


Wikipedia article about annualcreditreport.com 


Reputable article about annualcreditreport.com 


Search results for [clevelandclinic.org 
Wikipedia article about clevelandclinic.org 


Reputable article about the best hospitals in the 
U.S. 


Search results for [csmonitor.com 
-site:csmonitor.com 


Wikipedia article about The Christian Science 
Monitor 


Search results for [kernel.org —site:kernel.org 
Wikipedia article about kernel.org 


Search to find reputation information 

Search to find reviews 

Negative review on a business review page 1 
Negative review on a business review page 2 
Negative news article 


Search to find reputation information 
Negative review on a business review page 
Wikipedia article 

Article about business’ criminal behavior 


Negative review 1 
Negative review 2 
Negative review 3 
Negative review 4 
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Positive reputation information: Users in the U.S. 
can obtain free credit reports on this website by 
providing their Social Security Number. Note that 
the Wikipedia article tells us that 
“AnnualCreditReport.com is the only federally 
mandated and authorized source for obtaining a 
free credit report.” 


Positive reputation information: According to 
Wikipedia, the Cleveland Clinic “is currently 
regarded as one of the top 4 hospitals in the United 
States” which can be confirmed by reputable news 
articles cited in the references section. Users can 
trust medical information on this website. 


Positive reputation information: Notice the 
highlighted section in the Wikipedia article about 
The Christian Science Monitor newspaper, which 
tells us that the newspaper has won several 
prestigious awards. From this information, it can be 
inferred that the csmonitor.com website has a 
positive reputation. 


Positive reputation information: According to the 
Wikipedia article, “Kernel.org is a main repository of 
source code for the Linux kernel, the base of the 
popular Linux operating system. It makes all 
versions of the source code available to all users. It 
also hosts various other projects, like Google 
Android. The main purpose of the site is to host a 
repository for Linux kernel developers.” 


Extremely negative reputation information: This 
website appears to be a perfectly fine-looking store; 
however, much evidence of negative reputation can 
be found. The business has a very low rating on a 
business review site. There is a news article about 
financial fraud. There are many reviews on different 
websites describing users sending money and not 
receiving anything. 


Extremely negative/malicious reputation 
information: This website engaged in criminal 
behavior such as physically threatening users. 


Extremely negative reputation information: 
There are many detailed negative articles on news 
sites and charity watchdog sites about this 
organization describing fraud and financial 
mishandling. 
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3.3.4 Reputation of the Content Creators 


For individual authors and content creators, biographical information articles and online discussions can be a good source 
of reputation information. Expect to find more formal reputation information about people who create content in a 
journalistic, scientific, academic, or other traditionally professional capacity, as they often need online credibility for 
professional success. Educational degrees, peer validation, expert co-authors, and citations can be evidence of positive 
reputation information for professionals who publish their work. Employment history can also support a positive reputation 
for topics where training, credentials, or experience are important. 


Influencers and other individual content creators who earn income on social media platforms often have reputation 
information available as well, ranging from from biographical details or news articles to less formal reputation sources 
such as comments by other influencers in their fields. 


For non-professional content creators including ordinary people who post on social media or forums, you may find 
informal reputation information on the page itself such as comments by other people about the creators. For example, 
you may find comments or posts from other users helpful to see what other people think about a particular content creator. 


3.3.5 What to Do When You Find No Reputation Information 


You should expect to find some reputation information for large websites and well-known content creators. People or 
businesses who create content in a professional capacity typically have some reputation information available. 


However, small websites may have little or no reputation information. This is not indicative of high or low quality. Many 
small local businesses or community organizations have a small “web presence” and rely on word of mouth. 


Furthermore, many ordinary people participate in forum discussions or post on social media websites in a personal 
capacity. People may use an alias or username to post in order to avoid sharing personally identifiable information online. 
There may be no reputation information available for some individuals. A lack of reputation about people who post 
personal content is neither a positive nor a negative sign in your assessment of the page. 


Finally, remember that there are several important considerations involved in PQ rating (refer back to Section 3.1). 


Reputation is important, but if reputation information is not available for a website or content creator, pay extra attention to 
other PQ considerations, especially when assessing pages on YMYL topics. 
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3.4 Experience, Expertise, Authoritativeness, and Trust (E-E-A-T) 


Experience, Expertise, Authoritativeness and Trust (E-E-A-T) are all important considerations in PQ rating. The most 
important member at the center of the E-E-A-T family is Trust. 


Authoritativeness 


Experience 


Trust: Consider the extent to which the page is accurate, honest, safe, and reliable. 


The type and amount of Trust needed depends on the page, for example: 
e Online stores need secure online payment systems and reliable customer service. 
e Product reviews should be honest and written to help others make informed purchasing decisions (rather than 
solely to sell the product). 
e Informational pages on clear YMYL topics must be accurate to prevent harm to people and society. 
e Social media posts on non-YMYL topics may not need a high level of Trust, such as when the purpose of the 
post is to entertain its audience and the content of the post does not risk causing harm. 


Experience, Expertise and Authoritativeness are important concepts that can support your assessment of Trust: 


Experience: Consider the extent to which the content creator has the necessary first-hand or life experience for 
the topic. Many types of pages are trustworthy and achieve their purpose well when created by people with a 
wealth of personal experience. For example, which would you trust: a product review from someone who has 
personally used the product or a "review" by someone who has not? 


Expertise: Consider the extent to which the content creator has the necessary knowledge or skill for the topic. 
Different topics require different levels and types of expertise to be trustworthy. For example, which would you 
trust: home electrical rewiring advice from a skilled electrician or from an antique homes enthusiast who has no 


knowledge of electrical wiring? 


Authoritativeness: Consider the extent to which the content creator or the website is known as a go-to source 
for the topic. While most topics do not have one official, Authoritative website or content creator, when they do, 
that website or content creator is often among the most reliable and trustworthy sources. For example, a local 
business profile page on social media may be the authoritative and trusted source for what is on sale now. The 
official government page for getting a passport is the unique, official, and authoritative source for passport 
renewal. 


Experience, Expertise, and Authoritativeness may overlap for some page types and topics (for example, someone may 
develop Expertise in a topic due to first-hand Experience accumulated over time), and different combinations of E-E-A 
may be relevant to different topics. You should consider the purpose, type, and topic of the page, then ask yourself what 
would make the content creator a trustworthy source in that context. 
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Trust is the most important member of the E-E-A-T family because untrustworthy pages have low E-E-A-T no matter how 
Experienced, Expert, or Authoritative they may seem. For example, a financial scam is untrustworthy, even if the content 
creator is a highly experienced and expert scammer who is considered the go-to on running scams! 


When it comes to Page Quality rating, your assessment of E-E-A-T should be informed by one or more of the following: 

e What the website or content creators say about themselves: Look at the "About us" page on the website or 
profile page of the content creator as a starting point. Is the website or content creator a trustworthy source 
based on this information? 

e What others say about the website or content creators: Look for independent reviews, references, news 
articles, and other sources of credible information about the website or content creators. Is there independent, 
reliable evidence that the website or content creator is experienced, has expertise, is authoritative, or is otherwise 
considered trustworthy? Is there independent, reliable evidence that the website or creator is untrustworthy? 

e What is visible on the page, including the Main Content and sections such as reviews and comments: For 
some types of pages, the level of experience and expertise may be clear from the MC itself. What evidence can 
you gather from examining the MC or testing the page out? For example, you may be able to tell that someone is 
an expert in hair styling by watching a video of them in action (styling someone's hair) and reading others' 
comments (commenters often highlight expertise or lack thereof). 


Important: The website or content creator may not be a trustworthy source if there is a clear conflict of interest. For 
example, product reviews by people who own the product and share their experiences can be very valuable and 
trustworthy. However, "reviews" by the product manufacturer ("Our product is great!") or "reviews" from an influencer who 
is paid to promote the product are not as trustworthy due to the conflict of interest. 


Finally, there are many aspects of Trust, some which are not captured by Experience, Expertise and Authoritativeness. 


Please consider other aspects in your overall Trust assessment, such as customer service information for online stores or 
peer-reviewed publications for academic authors. If a page is untrustworthy for any reason, it has low E-E-A-T. 
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3.4.1 YMYL Topics: Experience or Expertise? 


Pages on YMYL topics can be created for a wide variety of different purposes. If the purpose of a page on a clear YMYL 
topic is to give information or offer advice, a high level of expertise may be required for the page to be trustworthy. 


However, sometimes pages on YMYL topics are created to share personal experiences, often regarding difficult life 
challenges. People turn to each other in times of need to share their own experience, seek comfort or inspiration, and 
learn from others. Factual information from experts and authoritative sources may not satisfy this need. 


Pages that share first-hand life experience on clear YMYL topics may be considered to have high E-E-A-T as long as the 


content is trustworthy, safe, and consistent with well-established expert consensus. 


information and advice must come from experts. 


Here are some examples: 


YMYL Topic 


Sleep challenges when 
pregnant 


(YMYL Health or Safety) 


Liver cancer treatment 
(YMYL Health or Safety) 
Filling out tax forms 
(YMYL Financial Security) 
Saving for retirement 


(YMYL Financial Security) 


How to vote 


(YMYL Society) 


Valuable sharing of life Experience 


Safe and non-medical tips and tricks for 
sleeping in the last trimester of pregnancy, 
provided by people who have personally 
struggled with this challenge, for example: 
how to use pillows to sleep comfortably in 
a position that is safe for babies 


A sincere and respectful forum discussion 
where people are describing how they're 
coping with liver cancer treatment 


A humorous video from a non-expert 
content creator about the frustration of 
doing taxes 


Reviews of retirement saving services by 
people with first-hand experience using 
these services 


A social media post by an ordinary citizen 
describing why they personally believe it's 
important to vote in local elections 


Copyright 2023 


In contrast, some types of YMYL 


Information or advice best left to Experts 


Sleep medications that are safe during 
pregnancy 


Different treatment options for liver cancer 
and the associated life expectancies 
under each treatment 


Instructions on how to fill out tax forms 


Advice on how to invest for retirement: 
how much to save, what kinds of assets to 
invest in, how much money you will need 
to retire by a specific age, etc. 


Information about who is eligible to vote or 
how to register to vote 
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4.0 Lowest Quality Pages 


Lowest quality pages are untrustworthy, deceptive, harmful to people or society, or have other highly undesirable 
characteristics. The considerations for Lowest quality include quality of the MC, reputation, E-E-A-T, etc., but there are 
special checks you will need to complete first. 


As a reminder, here are the steps for doing PQ rating: 


1. Assess the true purpose of the page. If the website or page has a harmful purpose or is designed to deceive 
people about its true purpose, it should be rated Lowest. 

2. Assess the potential of the page to cause harm as described in these guidelines. Websites or pages that are 
harmful to people or society, untrustworthy, or spammy as specified in these guidelines should receive the 
Lowest rating. 

3. Otherwise, the PQ rating is based on how well the page achieves its purpose using the criteria outlined in these 
guidelines. 


Steps 1 and 2 are a screening process to quickly detect harmful content before other PQ considerations are applied. 


To identify Lowest quality pages, start by considering the following: 


Initial Consideration Lowest Quality Pages 


The purpose of the page The Lowest rating is required if the page has a harmful purpose, or if 
it is designed to deceive people about its true purpose or who is 
responsible for the content on the page. 


The potential for the page to cause harm as The Lowest rating is required if the MC is harmful to self or others, 
described in these guidelines harmful to specified groups, or contains harmfully misleading 
information. 


The topic of the page, the type of website, and | A page on any topic or any type of website may qualify for Lowest. 
the extent to which YMYL standards apply 
Give special scrutiny to pages or websites needing a high level of 
trust, such as online stores, medical websites, or news coverage of 
major civic issues. 
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Next, assess the page to determine if any of the criteria for Lowest apply: 


Lowest Page Quality Assessment Any one of the following is justification for Lowest 


Quality of the MC e The page is hacked, defaced, or spammed. 
e The page is gibberish or otherwise makes no sense. 
e The MC is copied, auto-generated, or otherwise created without 
adequate effort. 
e The MC is created with so little effort, originality, talent, or skill 
that the page fails to achieve its purpose. 


The title of the page The page title is extremely misleading, shocking, or exaggerated. 
The role of Ads and SC on the page The MC is deliberately obstructed or obscured due to Ads, SC, 
interstitial pages, download links or other content that is beneficial to 


the website owner but not necessarily the website visitor. 


Information provided by the website and A complete lack of information about who is responsible for the 
content creator website and its content for YMYL pages or other pages requiring trust. 


Reputation of the website and content creator | A very negative reputation, including a reputation for malicious or 
harmful behavior. 


Trustworthiness of the page: E-E-A-T e The page or website is highly untrustworthy. 


e The lack of experience, expertise, authoritativeness or trust 
causes the page to fail to achieve its purpose. 
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4.1 Types of Lowest Quality Pages 


There are many different shapes and forms that Lowest pages can take. To help you keep track, the table below 
organizes Lowest pages into three broad categories: Harmful, Untrustworthy, and Spammy. These categories can 
overlap (e.g., pages that contain Harmfully Misleading Information are also Untrustworthy). 


Type of Lowest Page | Description Reference Section 
Harmful to Self or Pages that encourage, depict, incite or directly cause physical, mental, 
Other Individuals emotional or financial harm to self or other individuals. 

Examples include: Section 4.2 


e Detailed instructions on how to commit suicide 
e Detailed, realistic, and serious instructions written with the 
intent to help someone commit murder 


Harmful to Specified Pages that promote, condone, or incite violence or hatred against a 
Groups Specified Group of people (as defined in Section 4.3). 


Examples include: 
e Content that encourages violence or ill treatment towards a Section 4.3 
Specified Group 
e Content with extremely offensive/dehumanizing stereotypes of 
a Specified Group 


Harmfully Misleading | Pages that misinform people in a way that could cause harm. 


Information 
Examples include: 
e Clearly inaccurate harmful information that can easily be Section 4.4 
refuted by straightforward and widely accepted facts 
e Harmful unsubstantiated theories/claims not grounded in any 
reasonable facts or evidence 
Untrustworthy Pages or websites that are deceptive or have untrustworthy 
Webpages characteristics. 


Examples include: 


; : : i Section 4.5 
e Pages or websites with deceptive purpose or design me 
e Pages or websites designed to manipulate people into actions 
that benefit the website or other organization while causing 
harm to self, others or Specified Groups 
Spammy Webpages Pages with characteristics of webspam as defined in the Google 
Search Essentials Guidelines and/or Section 4.6 of these guidelines. 
Section 4.6 


Examples include: 
e Pages deliberately created with no MC or gibberish MC 
e Hacked, defaced, or spammed pages 


Important: There is lots of content that many people would find controversial, one-sided, off-putting, or distasteful, yet 
does not meet the criteria for Lowest as described in these guidelines. Use your judgment to follow the standards 
outlined throughout Section 4.0 rather than relying on personal opinions. 
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4.2 Harmful to Self or Other Individuals 


Use the Lowest rating for pages with content that encourages, depicts, incites, or directly causes harm to self or other 


individual 


S. 


Harm includes physical, mental, emotional, or financial harm to people. Pages should be considered Harmful to Self or 
Other Individuals if they directly attempt to harm people; encourage behavior that may result in harm; depict extremely 
violent or gory content without a beneficial/educational purpose; or otherwise are severely traumatic to people who view 


the page. 


Pages do not have to be harmful to all people to be considered Harmful to Self or Other Individuals. Different people have 
different levels of vulnerability to scams, awareness of potential dangers (e.g., dangerous feats depicted in stunt videos), 
and tolerance for viewing violent/disturbing content. If there is a reasonable possibility that viewing a particular page 
would cause harm to those who are most vulnerable, it should be considered harmful. 


Pages created with a beneficial purpose that report on, discuss, or inform about harmful actions or events (e.g., fictional 
entertainment, reputable news, education) should typically not be considered Harmful to Self or Other Individuals. For 
example, advocacy aimed at drawing attention to harmful, real-world actions or events (such as a webpage describing a 
protest against domestic violence) would not be considered Harmful to Self or Other Individuals. 


Here are some examples of content that is vs. is not considered Harmful to Self or Other Individuals: 


Harmful to Self or Other Individuals 


Incitement of violence towards Other Individuals 
Serious death threats or realistic-sounding threats 
Exposing personal information belonging to others 
with malicious intent to target them or promote 
harassment towards them (i.e., “doxxing”) 

How-to information that describes how to commit 
violent acts in an easy-to-replicate way 
Advocating for, glorifying, or trivializing violence 
and atrocities, or to disparage victim(s) of 
violence/atrocities 

Depicting or promoting information that facilitates 
or leads to serious harm to people, or discussions 
that attempt to justify abuse of people 
Encouraging unsafe behavior or substantially 
downplays the risks of dangerous activities (e.g., 
consuming household cleaning products) 

Suicide promotion or pro-anorexia content that 
encourages people to engage in behavior that can 
result in hospitalization or death 

Health-related advice that contradicts 
well-established expert consensus and could 
result in serious harm or could prevent someone 
from undertaking a life-saving treatment 
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NOT Harmful to Self or Other Individuals 


Depictions of violence in an action movie 

A news story about violent events 

Educational content that may depict violence or 
gross imagery 

An explanation of scams meant to raise 
awareness about them 

Portrayals of dangerous activities in a manner that 
discourages others from attempting the same feat 
(such as by clearly explaining the risks, describing 
the expertise and equipment required, etc.) 
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4.3 Harmful to Specified Groups 
Use the Lowest rating for pages that promote, condone, or incite hatred against a Specified Group of people. 


For the purposes of Search Quality rating, a Specified Group is a group of people that can be defined on the basis of: 
e Age (e.g., older adults) 

Caste (e.g., Dalits) 

Disability (e.g., people who are blind) 

Ethnicity (e.g., Roma) 

Gender Identity and Expression (e.g., transgender people) 

Immigration Status (e.g., student visa holders) 

Nationality (e.g., Italians) 

Race (e.g., Asians) 

Religion (e.g., Christians) 

Sex/Gender (e.g., men) 

Sexual Orientation (e.g., lesbians) 

Veteran Status (e.g., Marines) 

Victims of a major violent event and their kin (e.g., victims of the Holocaust) 

Any other characteristic that is associated with systemic discrimination or marginalization (e.g., refugees, people 

experiencing homelessness) 


The tone of the harmful content must be either serious (i.e., not joking or satirical) or mean-spirited (i.e., with an intent to 
demean or promote intolerance) to be considered Harmful to Specified Groups. Comedy or artistic expression related to 
a Specified Group should not be considered harmful unless it is clearly mean-spirited or has other clear harmful impact. 


Criticism of objects, philosophies, and ideas are generally not considered Harmful to Specified Groups. For example, 
negative criticism of a religious doctrine should not be considered targeted at the Specified Group that follows that 
religion. Remember that the content must promote, condone, or incite hatred of people to be considered Harmful to 
Specified Groups. 


Educational pages (e.g., definitions, research, academic papers), news stories, or other pages that have a beneficial 
purpose of informing society should not be considered Harmful to Specified Groups. Similarly, historical 
documents/videos that aim to capture the beliefs of different eras should not be considered Harmful to Specified Groups. 


Here are some examples of content that is vs. is not considered Harmful to Specified Groups: 


Harmful to Specified Groups NOT Harmful to Specified Groups 

e Encourages violence or ill treatment towards a e A historical documentary of WWII featuring 
Specified Group speeches from Nazi leaders 

e Promotes intolerance by demonstrating a staunch e A stand-up comedy routine that plays off of 
unwillingness to allow for the views, beliefs, or stereotypes in a way that is not mean-spirited 
behavior of a Specified Group e A newspaper article about a hate organization 

e Implies that one Specified Group is superior or e The dictionary definition of a slur 
inferior to another e A discussion about a particular religious text and 

e Contains extremely offensive/dehumanizing its views on women 


stereotypes of a Specified Group. Note that 
stereotypes can be negative or positive. 
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4.4 Harmfully Misleading Information 


Use the Lowest rating for pages that mislead people in ways that can cause harm to people and society. 

Misleading pages may have been produced with the intent to misinform people, or the content creator may believe that 
the inaccurate information they are sharing is true. There is an especially high standard for accuracy on clear YMYL 
topics or other topics where inaccurate information can cause harm. Be sure to research consequential facts or claims as 
necessary and to the extent the task time allows. 


Pages should be considered to contain Harmfully Misleading Information when they contain at least one of the following: 


Type of Harmfully Description 

Misleading Information 

Harmful and clearly Content that can be refuted by e False claims that a world leader died 
inaccurate information straightforward and widely accepted facts |e False dates for an election 


Harmful claims that contradict | Content that is inconsistent with positions, | e Claims that lemons cure cancer 
well-established expert facts, or findings that are widely agreed e Claims that buying lottery tickets is a 


consensus upon by authorities in the relevant field guaranteed way to save for retirement 


Harmful unsubstantiated Content that is not grounded in any e Claims that the 9-11 attacks were 
theories/claims reasonable facts or evidence, especially planned by the U.S. government 
those that could erode confidence in e Claims that world leaders are lizard 
public institutions. This includes people 
unsubstantiated theories that have been 
thoroughly debunked or are too outlandish 
to be given credence. 


Harmfully Misleading Information can occur from any websites or content creators - even seemingly "expert", 
"authoritative" or "official" ones. Any type of page with Harmfully Misleading Information should be rated Lowest, 
regardless of the source. 


However, note that some types of information are subjective, debatable, unverifiable, or inconsequential. For example, 
pages should not be considered to contain Harmfully Misleading Information if they exclusively contain: 

e Non-YMYL content created with a clear entertainment purpose, containing no hard claims of factual accuracy and 
are not harmful to people or society. Examples include many types of fiction, satire or parody, astrology, folklore, 
myths, and urban legends. 

e Reviews expressing personal preferences, opinions, or value-based judgments about a product, restaurant, 
book/movie/TV show, etc. 

e Claims or statements that are reasonably debatable when there is not a single established correct answer or truth 
(e.g., discussions about the relative effectiveness of different healthcare systems) 

e Insignificant errors or inaccurate information about a trivial topic (e.g., inaccuracies in the height of a celebrity) 


Pages that aim to persuade others that a certain position or perspective is correct are fairly common on the Internet. 
Pages with one-sided/opinionated/controversial/polarizing content should not be considered to contain Harmfully 
Misleading Information unless they could create harm to individuals or Specified Groups (as described above) and contain 
clearly inaccurate information, contradict well-established expert consensus, or are not grounded by reasonable 
facts/evidence. 


A webpage may be considered to have Harmfully Misleading Information based on the MC or on other characteristics of 
the page or information about the content creator (e.g., the title of an article is harmfully misleading, even if the article on 
its own is not; a creator blatantly misrepresenting their medical credentials for a video on medical topics). 


Harmfully Misleading Information can be especially hard to identify because it may require research from outside sources. 
Reputable fact-checking websites can't always keep up with the volume of unsubstantiated theories/claims produced by 
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the Internet, and some theories may even claim that debunking information is inaccurate. You should attempt to find 
high-quality, trustworthy sources to check accuracy and seek out the consensus of experts if you are unsure. Please 
research theories and claims to the extent the task time allows. If a theory/claim seems wildly improbable and can't be 
verified by independent trustworthy sources, you should consider it unsubstantiated. 


4.5 Untrustworthy Webpages or Websites 


The Lowest rating should be used for pages or websites you strongly suspect are engaging in deceptive or malicious 
practices. 


Some untrustworthy pages are created to benefit the website or organization rather than helping people. Some 
untrustworthy pages may even exist to cause harm to people who engage with the page, such as scams or malicious 
downloads. 


Your assessment of untrustworthiness may be based on the content of the page, information about the website, 
information about the content creator, and the reputation of the website or content creator. 


Your assessment may also be based on a lack of critically important information. For example, any website involving 
financial transactions or sensitive information should have comprehensive information about who is responsible for the 
site and a way to contact the site if something goes wrong. If some aspect of a page or website makes you suspect 
deception or maliciousness, please look for information about the site. If you cannot find reputation information to confirm 
your suspicion, carefully explore the site. 


Sometimes a single page on an unknown website in isolation may seem odd but subsequent exploration shows no 
concern. However, if you see a pattern of what appears to be deception or manipulation or become concerned about your 
own Safety, please use the Lowest rating and leave the website immediately. 


Pages with the following characteristics should be considered Untrustworthy: 
e Inadequate information about the website or content creator for its purpose 
Lowest E-E-A-T or Lowest reputation 
Deceptive purpose, deceptive page design, or deceptive intent 
Deliberately obstructed or obscured MC 
Characteristics of scams, malicious downloads, or other harmful behavior 
Any webpage or website designed to manipulate people into actions that benefit the website or other organization 
while causing harm to self, others, or Specified Groups 


Important: Highly untrustworthy pages should be given the Lowest rating even if you are unable to "prove" the webpage 
or site is harmful. Because many people are unwilling to use a highly untrustworthy page, an untrustworthy page or 
website fails to achieve its purpose. 


4.5.1 Inadequate Information about the Website or Content Creator for the Purpose of the Page 


For pages that require a high level of trust, information about who created the content and who is responsible for the 
content is critical. 


As discussed in Section 2.5.3, we expect most websites to have some information about who (e.g., what individual, 
company, business, foundation, etc.) is responsible for the website and who created the MC and some contact 
information, unless there is a good reason for anonymity. As you will see in the examples below, many types of Lowest 
pages such as malicious downloads and scams typically have no information or fake information about who is responsible 
to prevent recourse by people who are harmed. 


Any site that handles personal, private or sensitive data must provide extensive contact information. This includes sites 
that ask users to create passwords, share personal information, or conduct financial transactions. 


YMYL pages or websites that handle sensitive data with absolutely no information about the website or content creator 
should be rated Lowest. 
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4.5.2 Lowest E-E-A-T and Lowest Reputation of the Website or Content Creator 


If the E-E-A-T of a page is low enough, people cannot or should not use the MC of the page. If a page on YMYL topics is 
highly inexpert, it should be considered Untrustworthy and rated Lowest. 


Use the Lowest rating if the website and content creator have an extremely negative reputation, to the extent that many 
people would consider the webpage or website untrustworthy. 


Here are some examples originally shown in Section 3.3.3. All pages on these websites should be rated Lowest because 
of the credible extremely negative or malicious reputation of these websites, making them untrustworthy. 


Note: These website examples may be considered Harmful in some form as well. Sometimes reputation research 
exposes information that may not meet the criteria for considering a website harmful but does cause it to be untrustworthy. 


Website Description 


Evidence of fraudulent behavior: There are many reviews on websites describing how users 
Site selling children’s jungle gym | sent money and did not receive anything in return. There is also a news article about this 
business engaging in financial fraud. 


Site selling products related to Evidence of criminal behavior: This website engaged in criminal behavior such as physically 
eyewear threatening users. 


Organization serving the Evidence of fraudulent behavior: There are many detailed negative articles on news sites and 
hospitalized veteran community charity watchdog sites describing fraud and financial mishandling about this organization. 


4.5.3 Deceptive Page Purpose and Deceptive MC Design 


Pages or websites are Untrustworthy if they have a deceptive purpose. These pages or websites superficially appear to 
have one purpose, but in fact exist for a different reason. 


All pages with a deceptive purpose and/or deceptive MC should be rated Lowest because pages that engage in 
deception are Untrustworthy. 


The following list includes some common types of deceptive purpose pages. However, no list of deceptive page types will 
be complete—deceptive websites continue to evolve as people and search engines figure out how they are being tricked. 


e A webpage or website that impersonates a different site (e.g., copied logo or branding of an unaffiliated site, 
URL that mimics another site's name, etc.) 

e Anon-satirical social network profile made by an impersonator 

e A webpage or website that looks like a news source or information page, but in fact has articles to 
manipulate users in order to benefit a person, business, government, or other organization politically, monetarily, 
or otherwise 

e Awebpage claims to offer an independent review or share other information about a product, but is in 
fact created to make money for the owner of the website without attempting to help users. For example, 
the MC may contain intentionally misleading or inaccurate information created with the sole purpose of getting 
users to click on monetized links or buy the product 

e Awebsite claims to be the personal website of a celebrity, but the website is actually created to make 
money for the owner of the website without the permission of the celebrity. For example, the page may 
have false testimonials for a product and is created for the sole purpose of getting users to click on monetized 
links or buy the product 

e A webpage with a misleading title or a title that has nothing to do with the content on the page. People 
who come to the page expecting content related to the title will feel tricked or deceived 

e A webpage or website with deceptive website information. For example, the website may misrepresent who 
owns the site, what the website purpose really is, how the content was created, how to contact the site, etc. 
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Another form of deception is through the design of the page. Some pages are deliberately designed to manipulate users 
to take an action that will benefit the owner of the website rather than help the user. 


Here are some common types of pages with deceptive MC design: 

e Pages that disguise Ads as MC. Actual MC may be minimal or created to encourage users to click on the Ads. 
For example, fake search pages (example) that have a list of links that look like a page of search results. If you 
click on a few of the links, you will see that the page is just a collection of Ads disguised as search engine results. 
A “search box” is present, but submitting a new query just gives you a different page of Ads disguised as search 
results. 

e Pages that disguise Ads as website navigation links. For example, fake directory pages (example) that look 
like a personally curated set of helpful links, possibly with unique descriptions. In reality, the links are Ads or links 
to other similar pages on the site. Sometimes the descriptions of the links are unrelated to the page. 


Take a good look at the page and use your judgment. If you believe the true purpose of the page is not what it appears to 
be, the page should be rated Lowest. If you believe the page was deliberately created with the primary purpose to 
manipulate users to click on Ads, monetized links, questionable download links, etc., rather than to help people, the page 
should be rated Lowest. If the page uses deception in any form even if not described here, the page should be rated 
Lowest. 


4.5.4 Deliberately Obstructed or Obscured MC 


Pages are untrustworthy if the MC is deliberately obstructed or obscured due to Ads, SC, interstitial pages, download links 
or other content that is beneficial to the website owner but not necessarily the website visitor. Attempts to manipulate or 
coerce users away from the MC is evidence of untrustworthiness. 


Here are some examples of pages with deliberately obstructed or obscured MC that should be rated Lowest: 
e Ads that continue to cover the MC as the website visitor scrolls down the page. The Ads are virtually impossible 
to close without clicking on the Ad 
Pop-ups that obscure the MC and cannot be closed without taking an action that benefits the website 
An interstitial page that attempts to coerce a download or click that does not benefit the website visitor 
Ads that push the MC down so far that many users would not notice the MC 
MC in white-on-white text or MC presented so that it is difficult for a human being to read 


Logins or paywalls on otherwise trustworthy sites should not be considered untrustworthy, deceptive, or coercive. Many 
high quality sites such as newspapers or magazines cannot exist without monetization through subscriptions and 
paywalls. Some types of MC should be login protected, such as pages with personal information online banking websites 
or private pages on social network websites. 


MC may be inaccessible for non-deliberate or non-coercive reasons, such as content that doesn't load or content that is 
unavailable in a country or region. These are not reasons for Lowest. 


Use your judgment to assess whether the page is designed to manipulate or coerce users into taking actions that benefit 
the website. If you are unsure, look at other pages on the same website, conduct reputation research, etc. 


Important: Remember that many websites need monetization to share content with users. The presence of Ads alone is 
not enough for Lowest. 
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4.5.5 Suspected Malicious Behavior 


It may be difficult to prove that a page causes harm without experiencing harm yourself. For this reason, you may use the 
Lowest rating if you strongly suspect a page is malicious or harmful even without having proof. Any of the following 
should be considered untrustworthy: 


e Pages or websites that you strongly suspect are scams (see these links for more information about Internet 
scams: Avoid Scams 1, Avoid Scams 2, Internet Fraud). 

e Pages that ask for personal information without a legitimate reason (for example, pages that ask for name, 
birthdate, address, bank account, government ID number, etc.). 

e Pages that appear to “phish” for passwords to Facebook, Gmail, or other popular online services. See here for 
information about “phishing” fraud. 

e Pages with links that you strongly suspect are malware downloads. 


Important: Do not continue to interact with the page if you observe these characteristics. 


If you do click on a link and encounter a malware warning from your browser or antivirus software (example), please do 
not continue to that page. You can assume that the page is malicious for the purpose of rating and use Lowest. 


Note that not all warnings your browser may display are related to malware (e.g., other types of browser warnings include 
certificate acceptance requests, content filtering warnings, etc.). If you are unsure or hesitant to continue to the page for 
any reason, you can release the task. 


4.6 Spammy Webpages 


Pages and websites should exist for the benefit of people who visit the website. There are many types of webpages that 
can benefit visitors, such as online shopping or banking services, videos offering entertainment, or personal pages sharing 
a perspective or experience. Some of these pages depend on advertising and monetization to maintain the website and 
create high quality content while still benefiting visitors. 


However, some pages are created to benefit the website owner or other organizations and with little to no attention paid to 
the experience of the people who visit. When such pages use deceptive techniques to appear near the top of search 
results, it may disincentivize the creation of high quality content by crowding out pages created with effort, originality, 
talent, or skill that is helpful for visitors. 


This section describes characteristics of soam or spam-like pages that should be rated Lowest. The Google Search 
Essentials Guideline is a good reference for non-website owners to learn more about different spam techniques. If you 
recognize any of these spam techniques on a page, please use the Lowest rating. 


You do not need to develop spam recognition expertise or use special spam identification tools for PQ rating. Please 
review this section for guidance on what to look for even without having such tools or expertise. 
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4.6.1 Cannot Determine a Purpose, No MC, Little MC, or Lowest Quality MC 


All pages should be created with sufficient MC quality so that the page can achieve its purpose. However, some pages 
are so lacking in MC that the purpose of the page itself is unclear or you may suspect the page is not actually created for 
people. 


If any of the following are true, the page should be rated Lowest: 
e Despite your best efforts, the purpose of the page cannot truly be understood because the MC is gibberish or 
otherwise unusable for people 
e The page deliberately has no MC 
e The page is deliberately created with so little MC that it fails to achieve any purpose 
e The page is deliberately created with such low quality MC that it fails to achieve any purpose 


4.6.2 Hacked, Defaced, or Spammed Pages 
A hacked or defaced website is a site that has been modified without permission from the website owner(s). 


Websites can become hacked, defaced, or filled with a large amount of distracting and unhelpful content from bad actors. 
Bad actors may add unrelated MC or SC or may modify existing MC on webpages to change the purpose of the page. 
These pages should be rated Lowest because they fail to achieve their original purpose. Responsible website owners 
should regularly check their websites for suspicious behavior and take steps to protect users. 


A common problem on forum pages or in the comments sections of pages is "spammed" comments: posts with unrelated 
comments that are not intended to help other people, but rather to advertise a product or create a link to a website. 
Frequently these comments are posted by a “bot” rather than a real person. If a specific page on a website has unrelated 
“spammed” comments, the page should be rated Lowest. 


4.6.3 Auto-generated MC 


Creating an abundance of content with little effort or originality with no editing or manual curation is often the defining 
attribute of spammy websites. One way to do this is to use "auto-generated" content. 


It is possible to create many pages or even entire websites by designing a basic template from which hundreds or 
thousands of pages are created, sometimes using content from freely available sources (such as an RSS feed or API). 
These pages are created with very little effort and have little to no editing or manual curation. There is little value when 
pages repackage freely available content with little attention to no attention to the experience of people visiting the page 
(see here for more information on automatically generated content). 


It is often easiest to recognize auto-generated content by clicking on multiple pages on a website and observing 
"templated" content. It may be difficult to recognize auto-generated content from a single page, so be sure to explore the 
website if you suspect the content is templated. 


Pages and websites made up of auto-generated content with no editing or manual curation, and no original content or 
value added for users, should be rated Lowest. 
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4.6.4 Copied MC with No Added Value 


Another way to create an abundance of content with little effort or originality is to copy content from another source. The 
word “copied” refers to the practice of “scraping” content, or copying content from other non-affiliated websites without 
adding any original content or value to users (see here for more information on copied or scraped content). 


Any type of content may be copied: text, images, video, etc. Images may be slightly cropped or edited to avoid detection. 
People may alter videos as well, or even make a video of a screen playing video! Copied content with no value added is 
Lowest quality. 


Sometimes content is "embedded" or included on the page from another source. There is nothing wrong with embedding 
content, but the Lowest rating applies if there is no significant original content or value added on the page. 


Likewise, social media posts may be reposted by an unrelated account. There is nothing wrong with reposting content, 
but the Lowest rating applies if there is no value added in the form of additional content, context, or discussion. 


Pages created solely with copied, embedded, or reposted content with little to no editing or manual curation are lacking in 
effort and originality. As with auto-generated content, these techniques can be used to create many pages or even a 
whole website. For this reason, it is often easiest to recognize pages created solely with copied, embedded, or reposted 
content by exploring many pages on the site. 


The Lowest rating is appropriate if all or almost all of the MC on the page (including text, images or videos) is copied, 
embedded, or reposted with little or no effort, originality, talent, skill, manual curation, or added value for users. Such 
pages should be rated Lowest, even if the page assigns credit for the content to another source. 


All of the following are considered copied content: 

e Content copied, reposted, or embedded exactly from an identifiable source. Sometimes an entire page is 
copied, and sometimes just parts of the page are copied. Sometimes multiple pages are copied and then pasted 
together into a single page. Content that has been copied, reposted, or embedded exactly is usually the easiest 
type of copied content to identify. 

e Content that is copied, but changed slightly from the original. This type of copying makes it difficult to find 
the exact matching original source. Sometimes just a few words are changed, or whole sentences are changed, 
or a “find and replace” modification is made, where one word is replaced with another throughout the text. 
Images may be cropped or videos may be segmented into shorter clips. These types of changes are deliberately 
done to make it difficult to find the original source of the content. We call this kind of content “copied with minimal 
alteration.” 

e Content copied from a changing source, such as a search results page or news feed. You often won't be 
able to find an exact matching original source if it is a copy of “dynamic” content (content that changes frequently). 
However, we will still consider this to be copied content. 


Important: Licensed or syndicated content should not be considered as “copied” (see here for more on web syndication). 
Examples of syndicated content in the U.S. include news articles by AP or Reuters. 


4.6.5 How to Determine if Content is Copied 


How do you determine whether all or most of the MC is copied? How do you identify the original source of the content? 
These things can be difficult to determine, but the following steps may help. 


1. Copy a sentence or phrase in the text. It may be necessary to try a few sentences or phrases from the page just 
to be sure. When deciding what sentence or phrase to copy, try to find a sentence or series of several words 
without punctuation, unusual characters, or suspicious words that may have replaced the original text. 


2. Search on Google by pasting the sentence or phrase (surrounded by quotation marks to search for an exact 


match) inside the Google search box. You may also try without quotation marks to search for more general 
matches. 
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Google 


"Dorothy uses the Golden Cap to summon the Winged monkeys" 4 


Google Search I'm Feeling Lucky 


Here are some sentences and phrases from this page about “The Wizard of Oz” film and how to search for 
them. 


Sentence or Phrase From a Sentence Query to Find the Sentence or Phrase 


Try the query both with and without quotation marks around 
A sentence on the page: the sentence: 

[ “All are convinced by Dorothy that the Wizard can help 
All are convinced by Dorothy that the Wizard can help them | them too” ] 
too [ All are convinced by Dorothy that the Wizard can help 

them too ] 


Try the query both with and without quotation marks around 


A phrase on the page: the sentence: 
[ “Dorothy uses the Golden Cap to summon the Winged 
Dorothy uses the Golden Cap to summon the Winged Monkeys” ] 
Monkeys [ Dorothy uses the Golden Cap to summon the Winged 
Monkeys ] 


Try the query both with and without quotation marks around 
A different phrase on the page: the sentence: 

[ “revealing an old man who had journeyed to Oz from 
revealing an old man who had journeyed to Oz from Omaha | Omaha long ago in a hot air balloon” ] 
long ago in a hot air balloon [revealing an old man who had journeyed to Oz from 

Omaha long ago in a hot air balloon ] 


3. Compare the pages you find that match the sentence or phrase. Is most of their MC the same? If so, does one 
clearly come from a highly authoritative source that is known for original content creation (newspaper, magazine, 
medical foundation, etc.)? Does one source seem to reasonably be the original? Does one source appear to 
have the earliest publication date, verified by sources like the Wayback Machine? 


Use your best judgment. Sometimes it is clear that the content is copied from somewhere, but you cannot tell what the 
original source is. Other times the content found on the original source has changed enough that searches for sentences 
or phrases may no longer match the original source. For example, Wikipedia articles can change dramatically over time. 
Text copied from old copies may not match the current content. If you strongly suspect the page you are evaluating is not 
the original source, consider it likely to be copied. 


Based on this older version of a Wikipedia article about “The Wizard of Oz,” which can be found using the Internet Archive 
Wayback Machine, we can confirm that the text in the table has been copied. Please note that searching for the above 
sentence and phrases will not lead you to a current Wikipedia article about “The Wizard of Oz,” which demonstrates how 
confirming that the MC of a page contains copied content can sometimes be difficult. 


Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no effort, 
manual curation, or added value for users. Such pages should be rated Lowest, even if the page assigns credit for the 
content to another source. 
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4.7 


Webpage/Content 


Examples of Lowest Quality Pages 


Lowest: Superiority of Specified 
Group 1 


This is the homepage of 
Stormfront, a white supremacist 
website. The site is known for 
propagating white nationalism, 
antisemitism, Holocaust denial, 
white supremacy, and 
Islamophobia. 


Lowest: Superiority of Specified 
Group 2 


This page describes a 
non-scientifically backed theory 
of superiority of a Specified 
Group. 


Lowest: Non-medical drug use 
advice 


This post encourages mixing an 
anti-anxiety drug with a 
recreational drug. 


Lowest: Eating disorder 
encouragement 


This page has posts that 
encourage people to lose weight 
in unhealthy ways. 


Harmful to Specified Groups 


e Promotes, condones, or incites hatred 
against a Specified Group of people 
e Lowest website reputation 


Harmful to Specified Groups, Harmfully 
Misleading Information 


e Promotes, condones, or incites hatred 
against a Specified Group of people 

e Contains unsubstantiated theories/claims 
not grounded in any reasonable facts or 
evidence 


Harmful to Self or Other Individuals 


e Encourages, depicts, incites, or directly 
causes physical, mental, emotional, or 
financial harm to self or others 


Harmful to Self or Other Individuals, 
Harmfully Misleading Information 


e Encourages, depicts, incites, or directly 
causes physical, mental, emotional, or 
financial harm to self or others 

e Contains information that contradicts 
well-established expert consensus 
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Lowest Quality Justification PQ Rating and Explanation 


| | | | 


Lowest Medium Highest 


Inciting hatred towards Specified Groups 
significantly harms members of those groups 
and society as whole. This page is on a YMYL 
topic. 


This website has an extremely negative 
reputation for promoting white supremacy, 
antisemitism, Holocaust denial, etc. 


| | | | 


Lowest Medium Highest 


Unsubstantiated claims about a Specified 
Group significantly harms members of that 
group and society as a whole. This page is on 
a YMYL topic. 


The purpose of the page is to promote the 
view of superiority of a group of people. 


This page makes unsubstantiated claims 
about the role of melanin in the human body. 


| | | | 


Lowest Medium Highest 


Potentially harmful use of drugs could 
significantly impact a person’s health. This is a 
YMYL topic. 


This post and other content on the page 
encourages dangerous behavior that could 
cause harm. 


| | | | 


Lowest Medium Highest 


Eating disorders and decisions to seek 
medical treatment significantly impact a 
person’s health. This is a YMYL topic. 


This post encourages harmful behavior. 


"Proana" or pro-anorexia is a movement to 
view anorexia as a lifestyle choice, not a 
disease. Proana and similar "thinspiration" 
pages contradict well-established expert 
consensus that anorexia is a life-threatening 
eating disorder. 
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Webpage/Content 


Lowest: Personal information 
collection scam 


Superficially, this looks like a 
payment page on a merchant 
site. 


However, the form asks for highly 
sensitive personal information 
such as social security number, 
drivers license number, bank 
account numbers, and ATM PIN. 


Lowest: Poor financial advice 
and possible scam 


This is an article suggesting 
clicking on a link to avoid paying 
debt. 


Lowest: Miley Cyrus is dead 
"news article" on a deceptive 
website 


This is a factually inaccurate 
article on a website called "Now 
8 News". It falsely claims Miley 
Cyrus is dead. 


Lowest: Crystals heal cancer 
potential scam 


This article explains how crystals 
can be used to help cure 
cancers. It has a prominent 
"Shop Crystals" link at the top. 


Harmful to Self or Other Individuals, 
Untrustworthy 


e Encourages, depicts, incites, or directly 
causes physical, mental, emotional, or 
financial harm to self or others 

e Characteristics of scams, malicious 
downloads, or other harmful behavior 


Untrustworthy 


e Inadequate information about the 
website or content creator for the 
purpose of the page 

e Lowest E-E-A-T 

e Deceptive page design 

e Characteristics of scams, malicious 
downloads, or other harmful behavior 


Harmfully Misleading Information, 
Untrustworthy 


e Contains clearly inaccurate information 
that can be refuted by straightforward 
and widely accepted facts 

e Deceptive page design 


Harmfully Misleading Information, 
Untrustworthy 


e Contains unsubstantiated theories/claims 
not grounded in any reasonable facts or 
evidence 

e Contains information that contradicts 
well-established expert consensus 

e Characteristics of scams, malicious 
downloads, or other harmful behavior 
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Lowest Quality Justification PQ Rating and Explanation 


Medium Highest 


Sharing highly sensitive personal information 
could significantly impact a person’s financial 
security. This page is on a YMYL topic. 


While this looks like a merchant site, the true 
purpose of this page is to collect information 
that could be used to commit identity theft or 
remove money from someone's bank account. 


| | | | 


Lowest Medium Highest 


Debt management significantly impacts a 
person’s financial security. This is a YMYL 
topic. 


This page has multiple Lowest quality 
characteristics. From the MC, it is evident that 
the content creator lacks financial expertise 
(inconsistent with well-established expert 
consensus on what to do for debt relief). 


Furthermore, the page appears to be designed 
to manipulate people into clicking one of 
several prominent links rather than helping 
people by providing sound financial advice. 
The promise of debt relief is a common form of 
manipulation and scam technique. 


Lowest Medium Highest 


This website is designed to look like a news 
site and yet has deliberate factually inaccurate 
information. There is no information on who is 
responsible for this content. There is no date 
for the news article or date of death, which 
would be a key part of any standard reporting. 
This page was designed to misinform or 
deceive. 


| | | 


Lowest Medium Highest 


Claims about cancer cures could significantly 
impact people’s decisions to seek medical 
treatment. This page is on a YMYL topic. 


The claims made about the healing power of 
crystals contradict well-established expert 
scientific and medical consensus. 


Selling "cures" for diseases such as cancer is 
a commonly used scam technique. 
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Webpage/Content 


Lowest: Flat earth 


This is a homepage of The Flat 
Earth Society, a group of people 
dedicated to spreading the belief 
that the earth is flat. 


Lowest: Dry socket medical info 


This YMYL medical site has 
literally no information about who 
is responsible for the content or 
the website. 


Lowest: Merchant with little 


contact info, untrustworthy 
characteristics 


This looks like a product listing 
page on a merchant website. 


Harmfully Misleading Information 


Contains unsubstantiated theories/claims 
not grounded in any reasonable facts or 
evidence 

Contains clearly inaccurate information 
that can be refuted by straightforward 
and widely accepted facts 

Contains information that contradicts 
well-established expert consensus 


Untrustworthy 


Inadequate information about the 
website or content creator for the 
purpose of the page 


Untrustworthy 


Inadequate information about the 
website or content creator for the 
purpose of the page 

Characteristics of scams, malicious 
downloads, or other harmful behavior 
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Lowest Quality Justification PQ Rating and Explanation 


| | | | 


Lowest Medium Highest 


Claims contradicting well-established scientific 
consensus about the shape of the earth 
undermine trust in scientific institutions such 
as NOAA, the agency responsible for issuing 
alerts during dangerous weather events. This 
is a YMYL topic. 


The purpose of this website is promoting 
misleading information. Itis harmful to people 
and society as it undermines trust in public 
institutions. 


i 


Lowest Medium Highest 


Medical information about this dental condition 
could significantly impact how people seek 
dental treatment. This page is on a YMYL 
topic. 


There is no information about who created this 
website, no contact information, and no 
information about the author. 


Medical pages require a high level of user 
trust. Because there is no information about 
who owns this website and who created this 
content, this is considered an untrustworthy 
website. 


Lowest Medium Highest 


Inaccurate information about shipping and 
returns, deceptive use of logos and no 
information about who is responsible make this 
shopping site appear potentially fraudulent and 
untrustworthy. 


This page is selling Nike Air Jordan shoes. 
However, there is inadequate contact 
information on the contact us page for a 
merchant site. 


In addition, the Shipping and Returns page 
has the name of another company that seems 
to be unrelated. There are also official looking 
logos at the bottom of the homepage, including 
the Better Business Bureau logo and Google 
Checkout logo, that don’t appear to be 
affiliated with the website. 
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Webpage/Content 


Lowest: Stomach flu lowest EAT 
article 


This poorly written article 


describes how to cure the 
stomach flu. 


Lowest: Pimple popping lowest 
EAT article 


This very poorly written article 
describes popping pimples. 


Lowest: Malicious merchant site 


This website appears to be an 
eyeglasses merchant site. 


Lowest: Chicken recipes no MC 


This looks like a list of chicken 
recipes, but is actually a list of 
links that are ads. 


Lowest Quality Justification PQ Rating and Explanation 


Harmfully Misleading Information, 
Untrustworthy 


e Contains information that contradicts 
well-established expert consensus 

e Inadequate information about the 
website or content creator for the 
purpose of the page 

e Lowest E-E-A-T 


Untrustworthy 


e Inadequate information about the 
website or content creator for the 
purpose of the page 

e Lowest E-E-A-T 


Harmful to Self or Other Individuals, 
Untrustworthy 


e Encourages, depicts, incites, or directly 
causes physical, mental, emotional, or 
financial harm to self or others 

e Lowest E-E-A-T 

e Characteristics of scams, malicious 
downloads, or other harmful behavior 


Untrustworthy, Spammy 


e NoMC 

Deceptive page design 

e Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 
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Lowest Medium Highest 


Many patient hospitalizations and deaths occur 
due to the flu each year. This topic could 
significantly impact a person’s health. This is a 
YMYL topic. 


The writing is poor and the information is not 
consistent with standard medical guidance. It 
is clear from the content the author does not 
have medical expertise. 


This page should not be trusted as a medical 
resource. 


| | | | 


Lowest Medium Highest 


The writing is extremely poor and there is 
virtually no information in this article. 


For example, the opening sentence is 
"Popping pimples could be or could be not the 
new trend of getting rid of them." 


It is clear from the content that the author does 
not have skin care expertise. 


l | | | | 


Lowest Medium Highest 


This website swindled people and physically 
threatened people who complained. These 
articles on Wikipedia and the New York Times 
describe the deceptive techniques used by this 
website and provide other negative information 
about the website and its owner. 


The reputation of this website is lowest based 
on reports of malicious behavior. 


Lowest Medium Highest 


MC should be created to benefit people who 
come to the website. Pages without MC 
should be considered spammy as they crowd 
out and disincentivize the creation of pages 
with high quality MC. 


Pages with Ads that are designed to look like 
MC should be considered untrustworthy. 
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Webpage/Content 


Lowest: Mesothelioma deceptive 
directory 


This page looks like a listing of 
resources about Mesothelioma, a 
form of lung cancer. 


Lowest: Ads deliberately obscure 
MC 


When you first visit the page, it 
appears that there is no MC, only 
Ads. If you scroll down, you will 
eventually find the MC at the 
bottom. 


Lowest: Deceptive and 
untrustworthy fake survey page 


This page appears to be a 
survey with the opportunity to win 
an Apple MacBook Air, an 
iPhone 5c, or a Galaxy S4. 


Untrustworthy, Spammy 


No MC 

Deceptive page design 

Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 


Untrustworthy 


Deliberately obstructed or obscured MC 


Untrustworthy, Spammy 


Inadequate information about the 
website or content creator for the 
purpose of the page 

Deceptive page design 

Characteristics of scams, malicious 
downloads, or other harmful behavior 
Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 
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Lowest Quality Justification PQ Rating and Explanation 


Lowest Medium Highest 


Information about cancer could significantly 
impact how people make decisions related to 
medical treatment. This page is ona YMYL 
topic. 


This page is designed to look like a directory 
page or a list of helpful resources, but every 
link is an Ad. There is no MC on this page. 


Pages without MC should be considered 
spammy as they crowd out and disincentivize 
the creation of pages with high quality MC. 


Pages that disguise Ads as directory listings or 
other MC should be considered untrustworthy. 


| | | | 


Lowest Medium Highest 


This page requires scrolling past so many Ads 
to get to the MC that many people may stop 
scrolling and leave the page before reaching 
the MC. 


Deliberately obscured MC to benefit the 
website with no help for people visiting the 
page should be considered untrustworthy. 


Lowest Medium Highest 


This page contains a fake survey, which 
appears to promise users the opportunity to 
win an Apple MacBook Air, an iPhone 5c, ora 
Galaxy S4. 


However, there is no information about who is 
responsible for the survey, even though upon 
clicking, the survey asks for highly personal 
information such as bank account numbers. 
The message at the top of the page, “Thank 
You from Goggle.com” and survey content 
could mislead users to think that the survey is 
affiliated with Google. 


This page is deceptive and should be 
considered untrustworthy. 
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Webpage/Content 


Lowest Quality Justification PQ Rating and Explanation 


Lowest: Lowest MC free credit 
report 


The title of the page suggests 
that it may provide a credit report 
from the government 
"freecreditreport gov". 


However, the page has gibberish 
MC and spammy looking 
features, such as ads or links 
surrounding the meaningless 
text. 


Lowest: Buy Xanax 


This is a page on the website for 
the U.S. Green Building Council. 
However, this page has the 
words "Buy Xanax" over and 
over again, along with related 
words about buying Xanax and a 
URL link. 


Untrustworthy, Spammy 


Extremely misleading title 

Inadequate information about the 
website or content creator for the 
purpose of the page 

Deceptive page design 
Characteristics of scams, malicious 
downloads, or other harmful behavior 
Cannot determine any purpose, e.g., 
gibberish MC 

Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 


Untrustworthy, Spammy 


Characteristics of scams, malicious 
downloads, or other harmful behavior 
Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 
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| | | | 


Lowest Medium Highest 


Credit reports could significantly impact one’s 
financial security. This is a YMYL topic. 


The MC is gibberish. There is no beneficial 
purpose of gibberish MC. The title of the page 
is misleading, given the MC. 


This page also has deceptive features in the 
supplementary content, such as a friend 
request, a prize alert, and a download button. 
It is highly likely that these are ads or 
malicious download links. 


| | | | 


Lowest Medium Highest 


Prescription drugs could significantly impact a 
person’s health. This is a YMYL topic. 


This specific page has been created or 
modified on this organization's website in 
hopes that people will click on the link to buy 
the prescription drug Xanax. 


This is a spam technique. Spammers hope 
that the page will rank well in search results 
because the website is otherwise high quality. 
The words "buy xanax" are used repeatedly in 
hopes that the page will show up for the 
search "buy xanax". 


Attempting to spam search engines and trick 


people into clicking on links is highly 
untrustworthy behavior. 
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Webpage/Content 


Lowest: Auto-generated car 
engine code 


This page is full of 
supplementary content, links, 
and large ads. 


The MC at the top gives 
information about a specific error 
code P2570 and a second block 
of main content in the middle 
shows more information about 
P2570. 


The MC is broken up by large 
ads and it is difficult to 
understand due to the poor 
content quality. 


Lowest Quality Justification PQ Rating and Explanation 


Untrustworthy, Spammy 


Inadequate information about the 
website or content creator for the 
purpose of the page 

Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 
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Lowest Medium Highest 


This page has low quality MC and spammy 
attributes. 


The page repeats the error code "P2570" 
beyond what is helpful for users- a technique 


known as irrelevant keywords. 


The small amount of MC is not original to the 
website. Lists of error codes are available 
either by purchase or by scraping. 


An important fact is missing at the top "When 
you check engine light came on code P2570 
the reason should be ." The reason is left out - 
this should be the most important content on 
the page! 


Looking at other pages on the website, it is 
clear that each page is created automatically 
by filling in a template from a list of error code 
information. There appears to be little to no 
manual editing or checking to make sure the 
error code content is correctly pulled in, as 
evident by the missing reason on this page. 


As with many spammy websites, there is no 
information about who is responsible for this 
website and no way to report problems with 
the page. 
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Webpage/Content 


Lowest: Auto-generated bulldog 
wrinkle wipes 


This page claims to list the best 


bulldog wrinkle wipes and uses 
affiliate links. 


Lowest: Viagra copied content 


This looks like an information 
website for the drug Viagra. 


Lowest Quality Justification PQ Rating and Explanation 


Untrustworthy, Spammy 


Inadequate information about the 
website or content creator for the 
purpose of the page 
Deceptive page design 

e Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 


Untrustworthy, Spammy 


Inadequate information about the 
website or content creator for the 
purpose of the page 
Deceptive page design 

e Content created with little to no effort, 
talent, skill, originality, manual curation, 
or added value for users 
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Lowest Medium Highest 


The MC on this page is copied with little or no 
time, effort, expertise, manual curation, or 
added value for users. 


Looking at the content at the bottom of the 
page and exploring other pages on the 
website, it is clear that each page is created 
automatically using content from Amazon and 
substituting different product names into a 
generic template. There appears to be little to 
no manual editing or curation on this page. 


The true purpose of this page is to bring users 
seeking information about products to this 
website in hopes that they will click and order 
products by clicking on Amazon affiliate links. 
This website does not have a beneficial 
purpose for users but rather was created to 
benefit the creator of the website. Though the 
website discloses that they use affiliate links, 
the site offers no added value for users. 


As with many deceptive websites, there is no 
information about who is responsible for this 
website. 


Lowest Medium Highest 


Information about prescription drugs could 
significantly impact a person’s health. This 
page is on a YMYL topic. 


All MC on this page is copied with little or no 
time, effort, expertise, manual curation, or 
added value for users. Here is the original 
source. 


Exploration of the pages on this website are 
similarly created with little expertise or value 
added for users. 


The true purpose of this page is to bring users 
seeking information about Viagra to this 
website in hopes that they will click on the 
“order now” link. The "order now" link goes to 
another website and is likely monetized. 
Therefore, this website does not have a 
beneficial purpose for users but rather was 
created to benefit another website. 


As with many deceptive websites, there is no 


information about who is responsible for this 
website. 
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5.0 Low Quality Pages 


Low quality pages do not achieve their purpose well because they are lacking in an important dimension or have a 
problematic aspect. Low quality pages do not meet the standards for Lowest but may have similar though milder 


undesirable characteristics. 


To identify Low quality pages, start by considering the following: 


Initial Consideration 
The purpose of the page 
The potential for the page to cause harm as 


described in these guidelines 


The topic of the page, the type of website, and 
the extent to which YMYL standards apply 


Low Quality Pages 

Low quality pages have a beneficial or non-harmful purpose. 

Low quality pages have either no potential to cause harm, or have 
some mild potential for harm but do not meet the standards for Lowest 
as described in Section 4.0. 

A page on any topic or any type of website may qualify for Low. 

Give special scrutiny to pages or websites needing a high level of 


trust, such as online stores, medical websites, or news coverage of 
major civic issues. 


Next, assess the page to determine if any of the criteria for Low apply: 


Low Page Quality Assessment 


Quality of the MC 


The title of the page 
The role of Ads and SC on the page 
Information provided by the website and 


content creator 


Reputation of the website and content creator 


Trustworthiness of the page: E-E-A-T 


Any one of the following is justification for Low 


MC is created without adequate effort, originality, talent, or skill 
necessary to achieve the purpose of the page in a satisfying way. 


The page title is slightly misleading, shocking, or exaggerated. 


The Ads or SC significantly distract from or interrupt the use of the MC 
as described in Section 5.3. 


Unsatisfying amount of website information or information about the 
content creator for the purpose of the page. 


Important: For personal content shared on social media platforms or 
forums, an alias or username is adequate. 


A mildly negative reputation of the website or content creator. 


Inadequate level of E-E-A-T for the purpose of the page. 


Important: Low quality pages can occur on any type of website, including academic websites, nonprofit websites, 
government websites, or any other generally helpful type of website. Low quality pages may be about any topic. Pages 
on YMYL topics need more scrutiny for signs of Low quality related to the MC and website/content creator because the 
page could impact a person’s health, financial stability, or safety, or the welfare or well-being of society. However, for any 
type of page, a single Low quality attribute is enough to use the Low quality rating. 
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5.1 Lacking E-E-A-T 


Low quality pages often lack an appropriate level of E-E-A-T for the topic or purpose of the page. Here are some 
examples: 


e The content creator lacks adequate experience, e.g. a restaurant review written by someone who has never eaten 
at the restaurant 

e The content creator lacks adequate expertise, e.g. an article about how to skydive written by someone with no 
expertise in the subject 

e The website or content creator is not an authoritative or trustworthy source for the topic of the page, e.g. tax form 
downloads provided on a cooking website. 

e The page or website is not trustworthy for its purpose, e.g. a shopping page with minimal customer service 
information 


Important: The Low rating should be used if the page lacks appropriate E-E-A-T for its purpose. No other considerations 
such as positive reputation or the type of website can overcome a lack of E-E-A-T for the topic or purpose of the page. 


5.2 Low Quality Main Content 


MC is Low quality if it is created without adequate effort, originality, talent, or skill necessary to achieve the purpose of the 
page in a satisfying way. Mild inaccuracies on informational pages are evidence of Low quality. However, if the page has 
harmful inaccuracies or so many inaccuracies that it feels untrustworthy, the Lowest rating is justified. 


There are many possible ways for MC to be Low quality, and Low effort is important to recognize in all forms. 


While you may not have experience creating MC on webpages, think about your experience with something similar: 
writing reports for school. Here are some common shortcuts that students often take to save effort when writing a report: 


Producing content quickly with little attention to important aspects such as clarity or organization. 

Saving time by summarizing or paraphrasing from other readily available sources. 

Filling a report with large pictures or other distracting content rather than clear, helpful information on the topic. 

Using images from other sources (magazines, stock image websites, etc.) to avoid having to create pictures or 

diagrams to support the material in the report. 

e Using commonly known facts that require little research, for example: “Argentina is a country. People live in 
Argentina. Argentina has borders.” 

e Using "filler" content, for example: "Pandas eat bamboo. Pandas eat a lot of bamboo. Bamboo is the best food for 

a Panda bear." 


Of course, webpages are more diverse than school reports, but you will recognize some of these same techniques on 
pages you encounter. Some Low quality content is created with little effort in order to have content to support 
monetization rather than creating original or effortful content to help users. "Filler" content may also be added, especially 
at the top of the page, forcing users to scroll down to reach the MC. 
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Here are some examples of pages with Low quality MC: 


Low Quality MC Examples 


A crafting tutorial page with instructions on how to make a 
basic craft and lots of unhelpful "filler" at the top, such as 
commonly known facts about the supplies needed or other 
non-crafting information. 


A list of the best vacuum cleaners with product images 
and reviews that are paraphrased or summarized from 
other sources, with minimal signs of effort or original 
content added by the content creator. 


A slide show of the top 10 world destinations using other 
people's pictures and existing top 10 lists. 


A page titled "How many centimeters are in a meter?" with 
a large amount of off-topic and unhelpful content such that 
the very small amount of helpful information is hard to find. 


A list of the tallest mountains in the world that is missing 
Mount Everest. 


Why the MC is considered Low quality 


Lacks adequate effort to clearly present the instructions to 
people hoping to make the craft. 


Lacks adequate effort and first-hand experience from the 
content creator. 


Lacks effort and originality. 


Lacks effort, has excessive filler, lacks depth, and has only 
commonly known facts. 


Lacks accuracy and effort. 


The title of the page is also considered part of the MC. Titles are important because they represent the page in search 
results and help users decide what to click on. A misleading title can lead to a poor search experience. Exaggerated or 
shocking titles are evidence of Low quality because of the poor user experience that results when users see the actual 


MC after reaching the page. 


Here are some examples of pages with Low quality titles: 


Low Quality Title Examples 


A page titled "Pink Elephant: Part 2 coming soon!", but 
includes only speculation and rumor about a potential 
movie sequel that may or may not be produced. 


A page titled "Is the World about to End? Mysterious 
Sightings of Sea Serpents Prompt Panic!", for an article 
about the remains of a dead fish on a beach. 


A page titled "Eat the Healthiest Foods in the WORLD to 
extend your life!!!" for a list of tips on how to add more 
fruits and vegetables to meals. 


Why the Title is considered Low quality 


Slightly misleading title would give people the impression 
that the movie sequel has a confirmed release date, and 
that the page would provide accurate information about it. 


Exaggerated and shocking title is a mismatch with the 
content of the article. 


Overly exaggerated title suggests that the page would 
offer something more than the basic nutrition advice it 
provides. 


The level of talent or skill required for artistic expression by ordinary people on social media or other websites that allow 
content sharing should be viewed leniently. In many cases, people post activities to share with friends, such as a video 
documenting a first attempt at a new hobby. These types of pages should not be considered Low on the basis of talent or 
skill because the primary purpose is sharing content. Use the Low rating only if the lack of talent or skill prevents the 


page from achieving its purpose well. 


Important: Low quality MC does not meet the standard for Lowest and is not harmful, but it is unsatisfying for the 
purpose of the page or causes the page to fail to achieve its purpose well. 
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5.3 Distracting Ads/SC 


We expect Ads and SC to be visible. However, some Ads, SC, or interstitial pages (i.e., pages displayed before or after 
the content you are expecting) make it difficult to use the MC. Pages with Ads, SC, or other features that significantly 
distract from or interrupt the use of the MC should be given a Low rating. 


A single pop-over Ad or interstitial page with a clear and easy-to-use close button is not terribly distracting, even though it 
may annoy some people. However, difficult-to-close Ads that follow page scrolls, or interstitial pages that require an app 
download, can be truly distracting and make the MC difficult to use. You can see examples of interstitial pages here. 


The content of the Ads, SC, or other features may be distracting as well: sexually suggestive images such as here, 
grotesque images such as here, and sexually suggestive Ads on otherwise non-sexual pages are very distracting. 


Finally, Ads and SC can be distracting if the titles or images of the Ads or SC are shocking or disturbing. Here is an 
example of a page with shocking and exaggerated titles, images, and text in the Ads and SC. 


Important: The Low rating should be used if the page has Ads, SC, or other features that significantly distract from or 
interrupt the use of the MC. 


5.4 Mildly Negative Reputation of the Website or Content Creator 
The Low rating should be used if the website or the content creator has a mildly negative reputation. 


If the MC was created by a content creator who is using the website to host it, research the reputation of the content 
creator. Expect to find reputation information on well-known journalists, authors, social media influencers, bloggers and 
vloggers, professionals such as lawyers and doctors, etc. Pay attention when there is evidence of mildly 
negative—though not malicious or financially fraudulent—reputation. However, a lack of reputation information for 
ordinary people and lesser-known content creators is expected and is not a sign of Low quality. 


Exercise care when researching the reputation of businesses. Try to find enough reviews to understand a range of 
customer opinions and experiences, and read the details of negative reviews and low ratings before inferring that the 
business overall has a negative reputation. A few negative customer service reviews are typical for businesses such as 
stores or restaurants. 


5.5 Unsatisfying Amount of Information about the Website or Content Creator 


Expect to find some form of website information for many or most websites. We expect clear information about who (e.g., 
what individual, company, business, foundation, etc.) created the MC, unless there is good reason for anonymity. A 
long-standing Internet alias or username can identify the content creator. For personal websites or forum discussions on 
non-YMYL topics, an email address or social media link alone may be sufficient. Many people are reasonably hesitant to 
share personally identifiable information online! 


Stores and websites that process financial transactions require a high level of user trust. If a store or financial transaction 
website has just an email address and physical address, it may be difficult to get help if there are issues with the 
transaction. Likewise, many other types of websites on clear YMYL topics also require a high level of user trust. 


Important: Pages that offer payment functionality or process other types of financial transactions should receive a Low 
rating if there is an unsatisfying amount of customer service information or contact information. Pages on YMYL topics 
and other pages that require a high level of user trust should receive a Low rating if there is an unsatisfying amount of 
information about who is responsible for the website or who created the content. For forums, social media pages, and 
other websites where people commonly exchange opinions, a username is an acceptable identifier. 
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5.6 


Type of Webpage/Content 


Low: News 1 


Article about nuclear power 


Low: News 2 


Article titled "Getting Rid Of 
Toxins After The Holiday Season 


Low: Informational 


40th birthday party food ideas 


Low: Article 1 


Article with tips for dressing for 
the office 


Examples of Low Quality Pages 


Low Quality Justification 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 

e MC lacks accuracy or does not align with 
well-established expert consensus 

e Low E-E-A-T for the purpose of the page 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 

e MC lacks accuracy or does not align with 
well-established expert consensus 

e Slightly misleading, shocking, or 
exaggerated title 

e Low E-E-A-T for the purpose of the page 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 

e Ads or SC significantly distract from or 
interrupt the use of MC 

e Low E-E-A-T for the purpose of the page 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 
e Low E-E-A-T for the purpose of the page 
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PQ Rating and Explanation 


Lowest Medium Highest 
Information about energy sources and nuclear 
power could significantly impact global 


industries and society. This is a YMYL topic. 


The writing of this article is unprofessional, 
including many grammar and punctuation 
errors. The MC also appears to have been 
paraphrased from a science article found ona 
different source, but with factual inaccuracies 
introduced throughout. The combination of low 
quality MC and lack of E-A-T justify the 
Lowest+ to Low rating. 


Lowest Medium Highest 


The title of the article is misleading and does 
not reflect the actual content of the page. The 
MC does not explain how to get rid of toxins or 
what the word "toxins" in the headline refers to. 


The content also has many problems: the 
writing quality is poor, and the article includes 
meaningless statements such as "water 
therapy is one of the easiest ways of beauty 
regimen since it will give enough moisture on 
the skin". The article fails to cite sources, and 
there is no evidence of E-A-T. 


Lowest Medium Highest 

This page and website have many 
characteristics of Low quality pages. Close 
observation shows MC that contains mostly 
commonly known information and poor quality 
writing, demonstrating a lack of effort. The MC 
is broken up by large Ads that significantly 
disrupt the user experience. 


Lowest Medium Highest 

This informational article has multiple Low 
quality characteristics leading to a Low or 
Lowest+ rating. There is a large amount of 
"filler" or meaningless content and a complete 
lack of effort in editing. In addition, the MC 
contains only commonly known information with 
no expertise or first-hand experience shared by 
the content creator. 
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Type of Webpage/Content 


Low: Article 2 


Using ginger for your health 


Low: Article 3 


How to adopt children from Iraq 


Low: Financial 


Page with advice on picking a 
quality stock for investment 


Low Quality Justification 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 

e MC lacks accuracy or does not align with 
well-established expert consensus 

e Low E-E-A-T for the purpose of the page 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 
e Low E-E-A-T for the purpose of the page 


e Low E-E-A-T for the purpose of the page 
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PQ Rating and Explanation 


Lowest Medium Highest 

This informational article has multiple Low 
quality characteristics leading to a Low or 
Lowest+ rating. There is garbled or 
meaningless content, a complete lack of effort 
in editing, and unrelated links in the middle of 
the page. Additionally, the article is on a health 
topic, and there are no signs that the content 
creator has relevant expertise or links to 
support the medical claims made in the article. 


Lowest Medium Highest 


Adopting a child significantly impacts the health 
and well-being of families. This is a YMYL 
topic. 


However, the steps listed here are merely 
commonly-known information that would be of 
little benefit to someone interested in adopting 
a child from Iraq. For example, step 1 says 
“Choose an adoption agency” and suggests 
looking in a phone book. This demonstrates a 
lack of effort in the creation of MC. 


There is almost no information specific to lraq. 
There is no evidence of expertise on adoption 
from the description about the author. 


Note that some Ads have the same format as 
links to other articles on this website, potentially 
making it a bit harder for users to visually 
distinguish these Ads from the MC and SC. 


Lowest Medium Highest 


Advice on stock investment could significantly 
impact a person’s financial decisions and 
security. This is a YMYL topic. 


The MC is superficial and includes mostly 
commonly known information for this topic, e.g. 
"look at key statistics" or "find companies with 
rising revenue". There is no evidence that the 
author has financial expertise. Because this 
article gives advice on a YMYL financial topic, 
lacking expertise is a reason for a Low rating. 
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Type of Webpage/Content 


Low: Medical 


Page with information about how 
long the flu lasts 


Low: Recipe 1 


Apple pie recipe 


Low: Recipe 2 


Gluten-free New York cheesecake 
recipe 


Low Quality Justification 


Low E-E-A-T for the purpose of the page 


Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 


Ads or SC significantly distract from or 
interrupt the use of MC 
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PQ Rating and Explanation 


Lowest Medium Highest 


Many patient hospitalizations and deaths occur 
due to the flu each year. This topic could 
significantly impact a person’s health. This is a 
YMYL topic. 


There is no evidence that the author has 
medical expertise. Because this article gives 
advice on a YMYL medical topic, lacking 
expertise is a reason for a Low rating. 


pj | | | 
Lowest Medium Highest 


This page and website have many 
characteristics of Low quality pages. The MC 
is low quality and lacks important information. 
For example, it gives no indication of how to 
make a crust and doesn’t list premade crust as 
an ingredient. It would be very difficult to 
actually make an apple pie using this recipe. 


This page is lacking the kind of helpful content 
we expect in recipe pages, such as user 
reviews or comments. Without such 
information, it’s hard to tell if the recipe is any 
good. The small pink text at the top is not 
helpful for users. 


Lowest Medium Highest 


This is an example of distracting and disruptive 
Ads/SC: there are large Ads in the main column 
pushing down the MC as well as highly 
distracting content on the right that are labeled 
“Top Posts & Pages.” It is unclear whether 
these are SC or Ads. Additionally, the images 
in the Ads/SC are shocking, racy, and 
completely unrelated to the MC, which creates 
a poor user experience. 


56 


6.0 Medium Quality Pages 


Medium pages have a beneficial purpose and achieve their purpose. There is nothing wrong with Medium quality pages. 
Expect to encounter many Medium quality pages in PQ rating tasks. 


To identify Medium quality pages, start by considering the following: 


Initial Consideration 
The purpose of the page 


The potential for the page to cause harm as 
described in these guidelines 


The topic of the page, the type of website, and 
the extent to which YMYL standards apply 


The title of the page 


The role of Ads and SC on the page 


Information provided by the website and 
content creator 


Medium Quality Pages 
Medium quality pages have a beneficial or non-harmful purpose. 


Medium quality pages are not expected to cause harm. 


A page on any topic or any type of website may qualify for Medium. 


Give special scrutiny to pages on YMYL topics or websites needing a 
high level of trust, such as online stores. 


Medium quality pages have titles that summarize the page. 


The Ads and SC do not block or significantly interfere with the MC on 
Medium quality pages. 


Remember: Many websites need monetization to share content with 
users. The presence or absence of Ads alone is not a consideration 
for PQ rating. 


Medium quality pages have adequate information about the website 
and content creator for the purpose of the page. For stores or 
websites that process financial transactions, examine the customer 
service information. 


Important: For personal content shared on social media platforms or 
forums, an alias or username is adequate. 


Next, assess the page to determine if the criteria for Medium apply: 


Medium Page Quality Assessment 


Quality of the MC 


Reputation of the website and content creator 


Trustworthiness of the page: E-E-A-T 


Medium Quality Criteria 


MC created with adequate effort, originality, talent, or skill such that the 
page achieves its purpose. 


Not especially positive nor especially negative: Reputation information 
found about the website or content creator is not concerning, but not 
positive enough to justify a higher rating. 


Adequate level of E-E-A-T for the purpose of the page. 


Copyright 2023 57 


6.1 Types of Medium Quality Pages 


There are two types of Medium quality pages: 


Type of Medium Page Description 


Nothing wrong, but nothing 
special 


Mixed, but with some 
redeeming qualities 


All of the Medium Page Quality considerations and criteria apply. 


The page or website has some signs of High quality (E-E-A-T, quality of the MC, 
positive reputation), but also has one sign or mild signs of Low quality. 


Important: For a page to receive a rating higher than Medium, it must meet the criteria described in the next sections on 
High and Highest page quality. For a page to receive a rating lower than Medium, it must meet the criteria for Low or 
Lowest page quality. Medium is a rating that applies to many pages you will encounter in PQ rating tasks: many pages 
have nothing wrong, yet do not represent the high or highest quality content that's available online. 


6.2 Examples of Medium Quality Pages 


Webpage’/Type of Content Medium Quality Justification 


Medium: Encyclopedia Nothing wrong, but nothing special 


Wikipedia article about baroque ° 
pearls 


Adequate MC: achieves the purpose of 
the page 


Medium. Humor Nothing wrong, but nothing special 


Page from a humorous site e Adequate MC: achieves the purpose of 


the page 


Medium: Entertainment 1 Nothing wrong, but nothing special 


Article about “Keeping Up with the | e 
Kardashians” show 


Adequate MC: achieves the purpose of 
the page 
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PQ Rating and Explanation 


Lowest Medium Highest 


This is a short Wikipedia article about baroque 
pearls, a fairly narrow topic. This page 
achieves its purpose but doesn’t display 
characteristics associated with a High rating, 
such as effort in the form of interesting and 
meaningful content. 


Lowest Medium Highest 


This page is from a humorous site that 
encourages users to post photos with mouths 
drawn on them. This page achieves its 
purpose but doesn’t display characteristics 
associated with a High rating, such as effort in 
the form of discussion, engagement, or 
comments from other users. 


Lowest Medium Highest 


This page is on a website dedicated to 
entertainment news. This page achieves its 
purpose but doesn't display characteristics 
associated with a High rating, such as effort in 
the form of interesting and entertaining content, 
original images, etc. 
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Webpage’/Type of Content 


Medium: Entertainment 2 


Article about Miley Cyrus 


Medium: Lyrics 


Song lyrics for the song “Never 
You/Fear Love” 


Medium: Recipe 1 


Mexi-Chicken Casserole on a 
newspaper website 


Medium: Recipe 2 


Recipe for cherry-topped cake 


Medium Quality Justification 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Adequate MC: achieves the purpose of 


the page 


Adequate MC: achieves the purpose of 


the page 


Adequate MC: achieves the purpose of 


the page 


Adequate MC: achieves the purpose of 


the page 
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PQ Rating and Explanation 


Lowest Medium Highest 


This page is from a news/entertainment 
website. This page achieves its purpose but 
doesn’t display characteristics associated with 
a High rating, such as effort in the form of 
interesting and entertaining content, original 
images, etc. 


| | p | 


Lowest Medium Highest 


There are many lyrics websites that have 
similar content. This page achieves its purpose 
but doesn’t display characteristics associated 
with a High rating, such as effort in the form of 
depth of content, originality, etc. 


Lowest Medium Highest 


This recipe was contributed by a cookbook 
author. However, the page has no SC related 
to the purpose of the page, such as reviews or 
links to other recipes, etc. This page achieves 
its purpose but doesn’t display characteristics 
associated with a High rating, such as effort in 
the form of original images or comments from 
users who followed the recipe. 


Lowest Medium Highest 


This website is known for high quality content 
about animals and the environment. This 
particular page has a recipe for kids. There 
isn’t much MC or SC. This page achieves its 
purpose but it doesn’t display characteristics 
associated with a High rating, such as effort in 
the form of more detailed steps or inspiration. 
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7.0 High Quality Pages 


High quality pages serve a beneficial purpose and achieve that purpose well. High quality pages exist for almost any 
beneficial purpose you can imagine, from providing information to selling products to making people laugh to artistic 


expression. 


To identify High quality pages, start by considering the following: 


Initial Consideration 
The purpose of the page 


The potential for the page to cause harm as 
described in these guidelines 


The topic of the page, the type of website, and 
the extent to which YMYL standards apply 


The title of the page 


The role of Ads and SC on the page 


Information provided by the website and 
content creator 


High Quality Pages 
High quality pages have a beneficial purpose. 


High quality pages are not expected to cause harm. 


A page on any topic or any type of website may qualify for High. 


Give special scrutiny to pages on YMYL topics or websites needing a 
high level of trust, such as online stores. 


High quality pages have titles that summarize the page. 


The Ads and SC do not block or significantly interfere with the MC on 
High quality pages. 


Remember: Many websites need monetization to share content with 
users. The presence or absence of Ads alone is not a consideration 
for PQ rating. 


High quality pages have adequate information about the website and 
content creator for the purpose of the page. For stores or websites 
that process financial transactions, examine the customer service 
information. 


Important: For personal content shared on social media platforms or 
forums, an alias or username is adequate. 


Next, assess the page to determine if the criteria for High apply: 


High Page Quality Assessment 


Quality of the MC 


Reputation of the website and content creator 


Trustworthiness of the page: E-E-A-T 


High Quality Pages must demonstrate at least one of the following 


MC created with a high level of effort, originality, talent, or skill such 
that the page achieves its purpose well. 


Positive reputation of the website for the topic of the page. 
Positive reputation of the content creator for the topic of the MC. 


High level of E-E-A-T for the purpose of the page. 


Copyright 2023 60 


7.1 High Quality Main Content 


High quality MC should be satisfying for people visiting the page. High quality MC shows evidence of effort, originality, 
talent, or skill. For informational pages, High quality MC must be accurate and consistent with well-established expert 
consensus when such consensus exists. Have high standards! If you aren't sure whether the content is high quality, try 
finding other pages on the same topic to help calibrate your assessment: "typical" and "average" pages on a topic 
generally have Medium (not High) quality MC. 


To help develop your standards, here are some examples of High quality MC in contrast with Low quality MC: 


High Quality MC: Details and Examples Low Quality MC: Details and Examples 


High level of effort: The website or content creators Low level of effort: The MC shows signs of a lack of 
worked hard to create content that achieves the purpose effort by the website or content creators. 
of the page. The MC is well-organized, edited, and 


curated to support the purpose. e Lack of curation or editing: Content that helps the 
page achieve its purpose is mixed with less helpful 
e A news article with accuracy, depth, and clarity distracting or filler content 
e A Q&A page or forum post with meaningful e Lack of content creation: A forum post with little 
discussion by multiple participants discussion or only superficial comments 
e A well-organized crafting tutorial page with clear, e Lack of organization: A crafting tutorial page with 
helpful instructions so that others can make the lots of unhelpful "filler" at the top and little effort put 
craft successfully into explaining how to make the craft (the purpose 


of the page) at the bottom 


High originality Low originality 
e The MC is unique or original to the website e Information is summarized from other sources with 
e Original photos or video footage produced by the little added value 
website or content creator e Photos or videos come from other sources 
e The content on the page is unique to the content e Asummary of the perspectives of others, such as 
creators, such as a personal perspective based on summarizing product reviews written by others 


first-hand life experience 


High level of talent or skill Low level of talent or skill 
e The MC showcases the talent of the creator, e.g. a e A lack of adequate talent or skill prevents the page 
video of a talented content creator dancing from achieving its purpose, e.g. a how-to article on 
e The MC allows the page to achieve its purpose well plumbing by someone who does not have the 
because the content creator has talent or skill, e.g. necessary skill to explain the steps accurately 


a how-to article on plumbing by a skilled plumber 


7.2 Positive Reputation 


Positive reputation of the website can be a reason for a High rating if the website is responsible for the MC. If the website 
is not responsible for the MC (e.g., posts on social media websites or forums), positive reputation of the content creator(s) 
can be a reason for the High rating. 


For non-YMYL topics, reputation information may be less formal. Popularity, user engagement, and user reviews can be 
considered evidence of reputation for non-YMYL websites: websites can be considered to have a positive reputation if 
they are popular and well-known for their topic or content type. 


Remember: Many smaller websites and ordinary people have little reputation information available. A page can still 
receive a High rating without reputation information. 
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7.3 High Level of E-E-A-T 


Depending on the purpose of the page, topic, and type of website, a high level of E-E-A-T may be required for the page to 
achieve its purpose well and be considered High quality. Pages with High E-E-A-T are trustworthy or very trustworthy. 


Experience is valuable for almost any topic. Social media posts and forum discussions are often High quality 
when they involve people sharing their experience. From writing symphonies to reviewing home appliances, 
first-hand experience can make a social media post or discussion page High quality. 


Expertise is required for satisfying content on a variety of topics, from hobbies such as photography to YMYL 
topics such as tax preparation. Think about the topic of the page and what expertise is needed to create 
satisfying, trustworthy content. There are many types of informal expertise that may be visible in the MC itself. 


Authoritative pages of all types can be found. Government tax websites are an authoritative source for tax 
forms. Local businesses and organizations may be go-to sources for local information. When looking at a 
page or website, consider whether it is considered a go-to, authoritative source for the type of information it is 
displaying. 


Trust is especially important for High quality pages that involve processing financial transactions or cover 
YMYL topics. Even if the topic is not YMYL, trust may still be required; for example, product reviews and pages 
offering advice require at least some level of trust. While not all pages require a high level of trust, a 
trustworthy page is often a satisfying one. 


Think about the topic of the page. What kind of E-E-A-T is required for the page to achieve its purpose well? 
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7.4 


Webpage’/Type of Content 


High: News 1 


Homepage of a newspaper 


High: News 2 


Opinion article on a newspaper 
website 


High: Government agency 


US Naval Observatory Master 
Clock page 


High: Humor 


Article on a humor website 


High: Small business 1 


Local fish & chips restaurant 


Examples of High Quality Pages 


High Quality Justification 


e Satisfying MC: achieves the purpose of 
the page well 

e Positive website reputation for the topic of 
the page 


e Satisfying MC: achieves the purpose of 
the page well 

e Positive website reputation for the topic of 
the page 

e High E-E-A-T for the purpose of the page 


e High E-E-A-T for the purpose of the page 


e Satisfying MC: achieves the purpose of 
the page well 

e Positive website reputation for the topic of 
the page 


e Satisfying MC: achieves the purpose of 
the page well 
e High E-E-A-T for the purpose of the page 
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PQ Rating and Explanation 


Lowest Medium 


p 


Highest 


The news on the homepage of a newspaper is 
often on topics that significantly impact society. 
This is a YMYL topic. 


This is the homepage of a newspaper that has 
won several prestigious awards and has a 
positive reputation for its objective reporting. 
The articles on the homepage are original 
reporting by professional journalists (effort, skill, 
and original content unique to this site). 


Lowest Medium 


jan] 


Highest 


This is an opinion piece written by the Editorial 
Board of the largest newspaper in the U.S. 
state of Minnesota. The page is clearly labeled 
as an opinion piece, welcoming a new football 
coach to the University of Minnesota. This is 
original content unique to this site that was 
created by a skilled editorial board with 
expertise in editorial opinion writing. 


Lowest Medium 


E, 


Highest 


The purpose of this page is to display the 
official US Naval Observatory Master Clock 
time in 7 different time zones. The page 
displays the clock information in a clear, 
easy-to-read format. The Naval Observatory is 
highly trustworthy and authoritative for this type 
of information. 


| | p | 


Lowest Medium Highest 


This website is well-known for its humorous, 
satirical articles. This is a cute example of a 
satisfying and funny article (talent, skill, and 
original content unique to this site). 


Lowest Medium 


n | 


Highest 


This is an “about us” page on a restaurant 
website. This page provides information on 
when the restaurant opened and what visitors 
can expect. Other pages on the website 
provide information about the restaurant 
including the address, menu, other contact 
information, etc. The website is the go-to 
source for information about itself 
(Authoritative). The MC is high quality (effort, 
original content unique to this site). 
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Webpage’/Type of Content 


High: Small business 2 


Local preservation center 


High: Blog post 


Parenting article about strollers 


High: Shopping 1 


Backpack shopping page on a 
popular store website 


High: Shopping 2 


Bathroom décor page on a store 
website 


High: Video 1 


Saturday Night Live video ona 
video website 


High Quality Justification 


e Satisfying MC: achieves the purpose of 


the page well 


e High E-E-A-T for the purpose of the page 


e Satisfying MC: achieves the purpose of 


the page well 


e Positive content creator reputation for the 


topic of the page 


e High E-E-A-T for the purpose of the page 


e Satisfying MC: achieves the purpose of 


the page well 


e Positive website reputation for the topic of 


the page 


e High E-E-A-T for the purpose of the page 


e Positive website reputation for the topic of 


the page 


High E-E-A-T for the purpose of the page 


e Satisfying MC: achieves the purpose of 


the page well 


e High E-E-A-T for the purpose of the page 
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PQ Rating and Explanation 


| | | p 


Lowest Medium Highest 


This is the News and Updates section of a 
local preservation center selling poultry, 
vegetables, and more (Expertise). The MC is 
high quality (effort, original content unique to 
this site). 


Note: This example was added in 2014 so the 
“News and Updates” were timely. 


| | | pj 


Lowest Medium Highest 


This is a blog post on a newspaper that has 
won several prestigious awards. The author of 
this blog post has become known as an expert 
on parenting issues (Expertise) and is a regular 
contributor to this and other media websites 
(positive content creator reputation). The MC 
demonstrates effort and is original to this site. 


| | | p 


Lowest Medium Highest 


The purpose of this page is to allow users to 
buy a school backpack. The page provides a 
lot of different backpack options, and some of 
them have user reviews. 


This is a well-known, reputable merchant, with 
detailed Customer Service information on the 
site, making it very trustworthy (Trust). The MC 
demonstrates effort, and the products on sale 
are unique to this site (effort, original content). 


| | | p| 


Lowest Medium Highest 


This company sells its own line of high end, 
fashionable baby and children’s furniture and 
accessories. It has a positive reputation and 
expertise in these specific types of goods. 
Many products sold on the site are unique to 
this company. 


| | | p 


Lowest Medium Highest 


The MC of this video page is an episode of 
Saturday Night Live, which represents original 
content produced by a talented organization 
with experience and expertise in producing 
humorous television shows. 
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Webpage’/Type of Content 


High: Video 2 


“An Engineer’s Guide to Cats” 
video 


High: Game 


Online game 


High: Encyclopedia 


Encyclopedia article about the 
American Civil War 


High Quality Justification 


e Satisfying MC: achieves the purpose of 
the page well 
e High E-E-A-T for the purpose of the page 


e Satisfying MC: achieves the purpose of 
the page well 


e Positive website reputation for the topic of 


the page 


e Satisfying MC: achieves the purpose of 
the page well 


e Positive website reputation for the topic of 


the page 
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PQ Rating and Explanation 


| | | p 


Lowest Medium Highest 


This is a humorous, high quality video, 
illustrating the proper care and practical 
benefits of cats (original content, talent). The 
two engineers in the video are very 
experienced in cat ownership (Experience). 
For a humorous video like this one, experience 
(rather than formal expertise in cat care) is 
enough for it to have high E-E-A-T. 


| | | p 


Lowest Medium Highest 


Highly engaging game with multiple levels that 
could entertain a child for hours (effort, original 
content, talent and skill in game design). This 
website has a positive reputation for educating 
children about animals, and this particular 

animal game is consistent with that reputation. 


| | | p 


Lowest Medium Highest 


This is a detailed article about the American 
Civil War. The citations support the E-E-A-T of 
this article. 


There is a large quantity of in-depth, original 
content (effort, accuracy). This website has a 
positive reputation as a good resource for 
finding a wealth of generally known facts about 
topics such as this one. 


Note: Although much of the content is visible on 
the mobile page, we consider the content under 
the headings (which you need to expand) to be 
part of the MC. (To see the screenshot, you will 
need to zoom in to the image.) 


65 


8.0 Highest Quality Pages 


Highest quality pages serve a beneficial purpose and achieve that purpose very well. The distinction between High and 
Highest is based on the quality of MC, the reputation of the website and content creator, and/or E-E-A-T. 


To identify Highest quality pages, start by considering the following: 


Initial Consideration Highest Quality Pages 
The purpose of the page Highest quality pages have a beneficial purpose. 
The potential for the page to cause harm as Highest quality pages are not expected to cause harm. 


described in these guidelines 
The topic of the page, the type of website, and | A page on any topic or any type of website may qualify for Highest. 
the extent to which YMYL standards apply 
Give special scrutiny to pages on YMYL topics or websites needing a 
high level of trust, such as online stores. 
The title of the page Highest quality pages have titles that summarize the page. 


The role of Ads and SC on the page The Ads and SC do not block or significantly interfere with the MC on 
Highest quality pages. 


Remember: Many websites need monetization to share content with 
users. The presence or absence of Ads alone is not a consideration 
for PQ rating. 
Information provided by the website and Highest quality pages have adequate information about the website 
content creator and content creator for the purpose of the page. For stores or 
websites that process financial transactions, examine the customer 
service information. 


Important: For personal content shared on social media platforms or 
forums, an alias or username is adequate. 


Next, assess the page to determine if the criteria for Highest apply: 


Highest Page Quality Assessment Highest Quality Pages must demonstrate at least one of the following 


Quality of the MC MC created with a very high level of effort, originality, talent, or skill 
such that the page achieves its purpose very well. 


Reputation of the website and content creator | Very positive reputation of the website for the topic of the page. 
Very positive reputation of the content creator for the topic of the MC. 


Trustworthiness of the page: E-E-A-T Very high level of E-E-A-T for the purpose of the page. 
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8.1 Very High Quality MC 


Very high quality MC should be highly satisfying for people visiting the page. Very high quality MC shows evidence of a 
high level of effort, originality, talent, or skill. For informational pages, very high quality MC must be accurate, clearly 
communicated, and consistent with well-established expert consensus when it exists. Very high quality MC represents 
some of the most outstanding content on a topic or type that's available online. 


The standards for Highest quality MC may be very different depending on the purpose, topic, and type of website. Here 
are some examples of Highest quality original content: 


e For news: Original reporting that provides information that would not otherwise have been known had the article 
not revealed it. Accurate, original, in-depth, and investigative reporting requires a high level of skill/talent and 
effort. Very high quality news content will include a description of primary sources and other original reporting 
referenced during the content creation process. Very high quality news content must be accurate and should 
meet professional journalistic standards. 

e For artistic content (videos, images, photography, writing, etc.): Unique and original content created by highly 
skilled and talented artists or content creators. Such artistic content requires a high level of skill/talent and effort. 
If the artistic content is related to a YMYL topic (e.g., artistic content with the purpose of informing or swaying 
opinion about YMYL topics), YMYL standards should apply. 

e For informational content: Original, accurate, comprehensive, clearly communicated, and should reflect expert 
consensus as appropriate. Expectations for different types of information may vary. For example, scientific 
papers have a different set of expectations than a social media post sharing information about a hobby such as 
stamp collecting. However, all types of very high quality informational content share common characteristics of 
accuracy and clarity of communication, in addition to meeting standards appropriate to the topic or field. 


The Highest rating may be justified for pages with very satisfying MC created with a very high level of effort, originality, 
talent, or skill. 


8.2 Very Positive Reputation 


Reputation research is important when giving Highest ratings. For YMYL topics, very positive reputation is often based 
on recommendations from known experts or professional societies appropriate to the topic of the page. Prestigious 
awards may be evidence as well, depending on the topic and type of content. 


For non-YMYL topics, reputation information may be less formal. Popularity, user engagement, and user reviews can be 
considered evidence of reputation for non-YMYL websites: websites and content creators can be considered to have a 
very positive reputation if they are seen as one of the best sources available online for a topic or type of content. 


Remember: Many smaller websites and ordinary people have little reputation information. A page can still receive a 
Highest rating without reputation information. 


8.3 Very High Level of E-E-A-T 


Very high E-E-A-T is a distinguishing factor for Highest quality pages. A website or content creator who is the uniquely 
authoritative, go-to source for a topic has very high E-E-A-T. A content creator with a wealth of experience may be 
considered to have very high E-E-A-T for topics where experience is the primary factor in trust. A very high level of 
expertise can justify a very high E-E-A-T assessment. Very high E-E-A-T websites and content creators are the most 
trusted sources on the internet for a particular topic. 


Think about what E-E-A-T means for the topic of the page. How important is first-hand experience? Who are the 


experts? What makes a source highly authoritative for the topic? What makes a website or content creator trustworthy 
for the topic? Standards for very high E-E-A-T will differ depending on the topic of the page. 
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8.4 


Webpage/Type of Content 


Highest: News 1 


Article on a newspaper website 


Highest: News 2 


Article on a newspaper website 


Highest: Informational 1 


Fact-checking page on an 
Internet rumor debunking website 


Examples of Highest Quality Pages 


Highest Quality Justification 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive content creator reputation 
for the topic of the page 

e Very high E-E-A-T for the purpose of the 


page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 
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PQ Rating and Explanation 


| | | | 
Lowest Medium Hignest 
Reporting on environmental toxicity can 
significantly impact the health and financial 
security of people, businesses and government 
agencies. This page is on a YMYL topic. 


This is a feature article ffom a newspaper that 
has won numerous awards (very positive 
reputation of the website). 


The page has a comprehensive amount of very 
high-quality MC, including in-depth reporting 
(effort, original content unique to this site, skill, 
accuracy) by authors with extensive knowledge 
and experience in investigative journalism 
(Expertise). 


| | | | 
Lowest Medium Hignest 
This is an article on a newspaper website that 
has won a variety of awards. 


The article has a comprehensive amount of 
very high quality MC that is unique and original, 
including in-depth investigative reporting by two 
authors with extensive knowledge and 
experience in investigative journalism (effort, 
original content unique to this site, skill, 
accuracy). The reporters were finalists for a 
prestigious news award for the investigative 
reporting of this article (very positive reputation 
of the content creators). 


| | | | 


Lowest Medium 


= 


Highest 


This page is on a well-known fact-checking and 
debunking website that covers urban legends, 
Internet rumors, and other stories of unknown 
or questionable origin (very positive reputation 
of the website). 


The page has a very satisfying amount of 
original MC for users seeking to validate a 
photo claiming to have discovered a mermaid 
(effort, skill in fact checking, original content). 
Users can trust the information on this page 
due to the website's positive reputation and 
high level of expertise in debunking stories of 
this type. 
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Webpage’/Type of Content 


Highest: Informational 2 


Ball gown wedding dress page 


Highest: Government agency 


Yosemite National Park 
information 


Highest: Technical 1 


Software tips 


Highest: Technical 2 


Linux Kernel archives 


Highest Quality Justification 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very high E-E-A-T for the purpose of the 
page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very high E-E-A-T for the purpose of the 
page 


e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 
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PQ Rating and Explanation 
| | | = 


Lowest Medium Highest 


This page has very satisfying MC for users 
interested in ball gown wedding dresses. An 
abundance of pictures, plus options to view by 
price range, style, etc., are part of what makes 
this page so satisfying (effort demonstrated in 
the depth and types of content). This page is 
on a very popular wedding planning website 
(very positive reputation of the website). 


| | | | 
Lowest Medium Hignest 
This page has very satisfying and helpful 
information about Yosemite National Park, 
including traffic alerts and links to other parts of 
the website with additional information (effort, 
accuracy). Since the page is on the official 
national park website, it is uniquely 
authoritative (highest E-E-A-T). 


| | | | 
Lowest Medium Hignest 
This page offers very satisfying and helpful MC 
for the purpose of the page. Because the page 
is on the official website of the company that 
produces the software, it is highly authoritative 
(highest E-E-A-T). 


| | | | 
Lowest Medium Hignest 
From this Wikipedia article, we learn that 
“Kernel.org is a main repository of source code 
for the Linux kernel, the base of the popular 
Linux operating system. It makes all versions 
of the source code available to all users... The 
main purpose of the site is to host a repository 
for Linux kernel developers and maintainers of 
Linux distributions.” 


This website is the most authoritative source for 
information about Linux (highest E-E-A-T) and 
has a very positive reputation in the topic of the 
page. 
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Webpage’/Type of Content 


Highest: Financial 1 


Credit report information website 


Highest: Financial 2 


Tax forms page 


Highest: Magazine Article 1 


Article titled “Secret Fears of the 
Super-Rich” 


Highest: Magazine Article 2 


Interview with musical artist 


Highest Quality Justification 


e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 


e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 
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PQ Rating and Explanation 


| | | | 
Lowest Medium Highest 
Credit reports significantly impact a person’s 
financial choices and opportunities. This is a 
YMYL topic. 


Users in the U.S. can obtain free credit reports 
on this website by providing their Social 
Security Number. This Wikipedia article tells us 
that this website is “the only federally mandated 
and authorized source for obtaining a free 
credit report". This website is uniquely 
authoritative (highest E-E-A-T) and has a very 
positive reputation for credit report information. 


| | | | 
Lowest Medium Highest 
Income tax forms significantly impact a 
person’s financial security. This isa YMYL 
topic. 


The purpose of the page is to provide income 
tax forms and publications. The website 
represents an agency of the U.S. federal 
government that handles taxes, so the page 
and forms are uniquely authoritative and 
trustworthy. 


| | | p 


Lowest Medium Highest 


This article is on a magazine website that has 
won multiple National Magazine awards (very 
positive reputation for this type of article). The 
article has in-depth MC that is unique and 
original (effort, skill, original content). 


Lowest Medium 


paj 


Highest 


This magazine is very well-known and well 
regarded for content about artists and bands, 
and specifically has a positive reputation for 
interviews with musicians. This article features 
in-depth, original content in its interview with 
the artist. 


70 


Webpage’/Type of Content 


Highest: Medical 1 


BMI calculator 


Highest: Medical 2 


Page about meningitis 


Highest: Medical 3 


Page about seasonal flu 


Highest: Medical 4 ° 
Health information on a hospital ° 
page 


Highest Quality Justification 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 
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PQ Rating and Explanation 


| | | | 
Lowest Medium Hignest 
BMI is used by healthcare professionals to 
screen for various health issues and can 
significantly impact a person’s medical 
treatment. This is a YMYL topic. 


The purpose of this page is to provide an online 
BMI (Body Mass Index) calculator. The 
calculator is functional and easy to use. This 
page was created with effort and skill, and it 
accomplishes its purpose very well by offering 
very high quality and satisfying MC. 


The website represents an institute that is part 
of the National Institutes of Health. It has an 
extremely good reputation and is an expert in 
medical topics. 


Lowest Medium Highest 
Information about meningitis could significantly 
impact a person’s health. This is a YMYL topic. 


This is a meningitis reference page ona 
trustworthy and authoritative website for a 
nonprofit medical research group. This website 
has a reputation of being one of the best web 
resources for medical information. 


Lowest Medium Hignest 

Many patient hospitalizations and deaths occur 
due to the flu each year. This topic could 
significantly impact a person’s health. This is a 
YMYL topic. 


This is an influenza reference page ona 
trustworthy and authoritative medical website. 
This website has a reputation of being one of 
the best web resources for medical information 
of this type. 


| | | | 
Lowest Medium Hignest 
Information about hospital services and 
treatments could significantly impact a person’s 
ability to seek and receive healthcare. This is a 
YMYL topic. 


According to this Wikipedia article, this hospital 
is “currently regarded as one of the top 4 
hospitals in the United States as rated by U.S. 
News & World Report.” Users can trust 
medical information on this website. 
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Webpage’/Type of Content 


Highest: Shopping 


Backpack shopping page 


Highest: Login 


Online banking login page 


Highest: Charity 


Homepage of a charity 


Highest: PDF File 


Campus map 


Highest: Recipe 


Chocolate chip cookie recipe on a 
recipe blog 


Highest Quality Justification 


Very satisfying MC: achieves the purpose 
of the page very well 

Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very positive website reputation for the 
topic of the page 

Very high E-E-A-T for the purpose of the 
page 


Very satisfying MC: achieves the purpose 
of the page very well 

Very high E-E-A-T for the purpose of the 
page 


Very satisfying MC: achieves the purpose 
of the page very well 
Very high E-E-A-T for the purpose of the 


page 
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PQ Rating and Explanation 
= 


Highest 


Lowest Medium 


The purpose of this page is to provide 
information about, and allow users to buy, a 
specific type of school backpack. The page 
provides a lot of helpful product information, as 
well as 600 user reviews (effort, original 
content). 


Since the store produces this backpack (unique 
product), they are experts on the product, 
making the page on their own website 
authoritative (highest E-E-A-T). In addition, this 
store has a reputation for producing one of the 
highest quality and most popular school 
backpacks on the market. 


Lowest Medium 


= 


Highest 


Accessing one’s financial information 
significantly impacts a person’s financial 
decisions and security. This is a YMYL topic. 


This page has login functionality and clear 
information about what the user is logging into. 
This is a large, popular bank that has a good 
reputation and is considered highly trustworthy. 


| | | p 


Lowest Medium Highest 


Disaster relief significantly impacts people and 
society in a time of crisis. This page is on a 
YMYL topic. 


This is a highly reputable charity according to 
multiple charity rating organizations. 


| | | p 


Lowest Medium Highest 


This PDF file is a detailed campus map of a 
major university, which is hosted on the official 
university website. This is a highly authoritative 
source for this information. The map includes a 
listing of all the buildings, parking structures, 
parking lots, construction areas, etc. (effort, 
skill, accuracy). 


Lowest Medium 


= 


Highest 


The author of this blog has documented her 
extensive experimentation with a chocolate 
chip cookie recipe (Experience), and her 
expertise and skill is demonstrated in the large 
quantity of unique, original, and very satisfying 
MC. 
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Webpage’/Type of Content 


Highest: Video 1 


The band OK Go’s music video 


for the song “This Too Shall Pass” 


Highest: Video 2 


“Henri 2, Paw de Deux” video 


Highest: Video 3 


Saturday Night Live video on the 
TV network’s website 


Highest Quality Justification 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive website reputation for the 
topic of the page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Very positive content creator reputation 
for the topic of the page 

e Very high E-E-A-T for the purpose of the 
page 
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PQ Rating and Explanation 
| | | | 


Lowest Medium 


= 


Highest 


This is a high quality, professionally produced 
video on the band’s official channel. The music 
video represents unique and original content 
created by a highly skilled and talented musical 
artist (talent, skill, effort, original content). 


In addition, this video sharing website has a 
positive reputation for hosting official music 
videos from bands. 


Lowest Medium 


= 


Highest 


This is an amateur video that went viral, 
created by a film school student about his 
mother's cat. It won the “Golden Kitty Award” 
for “Best Cat Video On The Internet” at 

the Walker Art Center's Internet Cat Video Film 
Festival. The video is unique and original 
content created with a high level of effort. 


In addition, this video sharing website has a 
positive reputation for cute cat videos. 


| | | p 


Lowest Medium Highest 


The MC of this video page is an episode of 
Saturday Night Live on the TV network’s official 
website, which is a go-to source for this 
content. The episode represents high quality 
and original content created by a TV show that 
has won numerous awards. Below the main 
video, there are many other videos that users 
may be interested in. 
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9.0 Page Quality Criteria for Specific Types of Pages 
9.1 Ratings for Encyclopedia Pages 


There are many encyclopedia-type websites. Some are highly-respected publications that are standard references, while 
some are websites with content created and edited by anonymous users, with no editorial oversight or fact checking. We 
may not always know the author of the specific encyclopedia article, and therefore must rely on website reputation 
research to determine the E-E-A-T of the article. High and Highest quality ratings should only be used for encyclopedias 
with very good reputations for accuracy and expertise, where the article itself is well-researched with appropriate 
references cited. 


A note about Wikipedia: in general, the website has a good reputation and is a very popular resource that is generally 
valued for accuracy. However, there is no single author or organization that vouches for the accuracy of Wikipedia 
articles, and the quality of pages varies. You should check individual articles to evaluate the quality of the MC. 


A Wikipedia article on a non-YMYL topic (example) with a satisfying amount of accurate information and trustworthy 
external references can usually be rated in the High range. A PQ rating in the Medium range is often appropriate for 
pages with less extensive MC and external references. Naturally, Wikipedia articles with very little MC should get lower 
PQ ratings. Factual inaccuracy is a sufficient reason for a Low or Lowest rating. 


9.2 Ratings for Pages with Error Messages or No MC 


In PQ rating tasks, you may encounter pages with error messages or other types of "broken" pages. Please think about 
whether the page offers help for users—did the website owner make an effort to ensure that users visiting the page have 
a good experience and get help finding what they are looking for? 


Pages with an explicit error (or custom 404) message are often Medium quality, since they tend to clearly communicate 
that there's a problem and help users navigate to other/better pages on the website. A rating of High (or occasionally 
even Highest) may be used for the rare error message page that involves a high level of effort and original content and 
provides a truly satisfying experience for users. 


Pages that are "broken" are often Low quality, because the broken aspect prevents them from achieving their purpose 
even if the problem is temporary and likely to be fixed later on by the website owner. However, sometimes "broken" 
pages should be rated Lowest if the problem is not temporary and instead is due to a systemic problem with the website. 
To differentiate between Low quality (temporarily broken) vs. Lowest quality (evidence of a systemic problem) pages, 
check other pages on the same website. If many other pages have the same problem, there is a systemic lack of effort 
on the part of the website owner that justifies the Lowest rating. Otherwise, use the Low rating to reflect the (hopefully 
temporary!) issue with the MC on the page. 


Here are some examples: 


Webpage’/Type of Content Characteristics of the Page PQ Rating and Explanation 
| | | | 
Lowest Medium Highest 
e NoMC or F 
Lowest: Deliberately created with e Content created with little to no effort, This en example of a page wit h no MC. You 
: s Pei : might think that the MC is “missing” due to a 
no MC (systemic problem) talent, skill, originality, manual curation, or 


problem with this particular page, but in fact, 
this website has hundreds of pages that look 
the same—no MC, just Ads. This website 
shows Ads with little or no attempt to help 
users, and should be rated Lowest quality. 


added value for users 
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Webpage’/Type of Content Characteristics of the Page 


e No MC (probably due to a temporary 
technical error) 
e Positive website reputation 


Low: Page didn't fully load and 
has no MC (temporarily broken) 


Medium: Error page with custom 
404 message 


Nothing wrong, but nothing special 


Hidhe rear pend wih cuter HOA Satisfying MC: achieves the purpose of 
eee the page well 


message e Positive website reputation 


Highest Error page withieustom Very satisfying MC: achieves the purpose 
Hanesi Error page win cusion, of the page very well 


g =: š i 
404 message e Very positive website reputation 
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PQ Rating and Explanation 


Lowest Medium Highest 


This page has no MC and no error message. It 
is an isolated example of a page with no MC or 
error message on a website for a reputable 
newspaper for a town in Michigan. All of the 
navigation links work, and the page was later 
fixed. This page can be viewed as lacking in 
effort (an error that was not caught before the 
page went live). 


Lowest Medium Highest 


This is an example of a “custom 404” page, 
alerting users that the URL they are trying to 
visit no longer exists. Some websites do a nice 
job of alerting users about a problem and 
providing helpful tips. 


This page is on a well-known merchant website 
with a good reputation. However, this particular 
page displays the bare minimum of content 
needed to explain the problem to users, and 
the only helpful content is a link to the 
homepage. 


| | | pj 


Lowest Medium Highest 


This is an example of a “custom 404” page, 
alerting users that the URL they are trying to 
visit no longer exists. This website does a nice 
job of explaining the issue and providing helpful 
tips, including a search box. 


| | | p 


Lowest Medium Highest 


This is an example of a “custom 404” page. 
These pages are designed to alert users that 
the URL they are trying to visit no longer exists. 
The MC of this page is the cartoon, the caption, 
and the search functionality, which is specific to 
the content of the website. Itis clear that effort 
and talent was involved in the creation of the 
MC. 


This publication has a very positive reputation 
and is specifically known for its cartoons, which 
allows us to go as high as High+ to Highest. 
This is among the most clever, original, and 
satisfying custom 404 messages that can be 
found online! 
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9.3 Ratings for Forums and Q&A Pages 
Ratings for forum and Q&A pages can be challenging. Here is some guidance on how to approach these pages: 


e The Main Content includes the question, the answers/responses, and the resulting discussions. 

Rate from the point of view of a user who visits the page, rather than a participant involved in the discussion. 

e Users who post answers/responses or comments are often identified only by a username or alias. A page 
can be High or Highest quality with just usernames or aliases depending on other criteria. 

e The E-E-A-T of a discussion among users can often be judged by the posts or comments themselves. 

o For some topics, Experience is the most important dimension of Trust. For other topics, assessing 
Expertise through the posts may be important. In some cases, the posters themselves will highlight either 
their own Experience or Expertise, or other people will comment on it. 

o Pages on YMYL topics require more attention to Trust and more care in the assessment of E-E-A-T. 

e High and Highest quality forum/Q&A pages have satisfying conversations that involve many participants, some 
of whom put a great deal of effort into their posts and have a wealth of Experience and/or Expertise on the topic. 
Such conversations can be very satisfying because of the depth of discussion, the unique insights, or the sharing 
of experiences that many would not have access to in their real-world community. 

e Low quality forum/Q&A pages often lack effort (few responses, surface-level rather than in-depth discussion), 
lack Experience or Expertise, contain mild inaccuracies, or show a significant lack of respect or decorum among 
the participants that might deter others from joining the discussion. 

e Lowest quality forum/Q&A pages may contain information or advice that is harmfully misleading, contradicts 
well-established expert consensus, encourages harm towards self or other individuals/participants, etc. 


Here are some examples: 


Webpage/Type of Content Characteristics of the Page PQ Rating and Explanation 
— | | | | 
Lowest Medium Highest 


Information about chest pain and smoking 
could significantly impact people’s decisions to 
seek medical treatment. This page is on a 
YMYL topic. 


This page must be evaluated from the point of 


e Harmfully Misleading Information: view of a user visiting this page from a search 


Lowest: Q&A page about chest Contains inaccurate and potentially engine, rather than a participant. The question 
pains and smoking dangerous medical advice about a YMYL |ÍS Poorly worded and difficult to understand. 


There aren't many responses. One response 
correctly warns about the danger of smoking 
cigarettes but suggests smoking marijuana 
instead. Another response suggests that it's 
fine to keep smoking and that aspirin can be 
used to address pain. The last response lacks 
respect: it asks if the person who asked the 
original question is stupid. 


topic 


The answers have incorrect and potentially 
dangerous medical advice, making it lowest 
quality MC. 
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Webpage’/Type of Content 


Low: Q&A page about a 2002 
Volvo part 


Low: Q&A page about a game 
console 


Medium: Forum page on an 
online auction website 


Medium: Forum page on how to 
wash ballet shoes 


Characteristics of the Page 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 
e Misleading page design 


e Unsatisfying MC for the purpose of the 
page: does not achieve the purpose well 


Mixed, but with some redeeming qualities: 
e High E-E-A-T for the purpose of the page 
e Unsatisfying MC for the purpose of the 

page: does not achieve the purpose well 


Mixed, but with some redeeming qualities: 
e Adequate MC: achieves the purpose of 
the page 
e Adsor SC significantly distract from or 
interrupt the use of MC 
e High E-E-A-T for the purpose of the page 


Copyright 2023 


PQ Rating and Explanation 


| =j | | | 


Lowest Medium Highest 

Please read the MC (areas with red boxes 
around it), including the completely unhelpful 
"answer." This answer is so unhelpful, we can 
consider this question to be unanswered. This 
page has an unsatisfying amount of MC. 


In addition to a very unhelpful “answer,” the 
page design makes it difficult to distinguish the 
MC from Ads. For example, below the answer, 
we see a "sponsored answer," which has the 
same format as the real answer, but is actually 
an Ad and not an answer to the question—this 
is a misleading page design. 


Lowest Medium Highest 


Some websites rely on users to create virtually 
all of their MC. In this case, the MC is the 
user’s question. If there are no answers, there 
is a lack of effort (since no one is participating 
or engaging by answering the user's question!). 
This page does not achieve its purpose well. 


Lowest 


pn | 


Medium Highest 


The relative lack of effort/participation in the 
discussion and depth of the MC is balanced by 
the high E-E-A-T of this forum. This forum is 
dedicated to this kind of question and therefore 
has a high level of expertise. 


Lowest 


p | 


Medium Highest 


This forum is about dance topics, and many 
pages have expertise from a community of 
ballet dancers. On this particular page, 
participants have first-hand experience washing 
ballet shoes and make recommendations 
based on their own experiences. 


This page is “mixed” because there is some 
distracting content that makes it hard to read 
the MC. However, this is not a beauty contest! 
Even though the page may be cluttered, there 
is some valuable E-E-A-T and helpful MC, 
making Medium a good rating for this page. 
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Webpage’/Type of Content 


High: Forum post on the magic 
loop technique in knitting 


High: Q&A page about whether a 
Roomba will work 


High: Forum page on 
authenticating a purse 


High: Forum page on landscaping 
an aquarium 


Characteristics of the Page 


MC is satisfying for the purpose of the 
page: demonstrates effort, originality, 
talent, or skill 

High E-E-A-T for the purpose of the page 


Satisfying MC: achieves the purpose of 
the page well 
High E-E-A-T for the purpose of the page 


Satisfying MC: achieves the purpose of 
the page well 

Positive website reputation for the topic of 
the page 

High E-E-A-T for the purpose of the page 


Satisfying MC: achieves the purpose of 
the page well 
High E-E-A-T for the purpose of the page 
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PQ Rating and Explanation 


| | p | 


Lowest Medium Highest 


The person who posted the first message on 
this forum page provides a helpful resource on 
how to master the magic loop technique in 
knitting (effort, skill). She also shares pictures 
of her own version using an old pair of blue 
jeans (original content). With over 20 years of 
experience knitting socks, we would consider 
her to be an expert on the topic. 


| | p | 


Lowest Medium Highest 


Many participants share their personal 
experiences with these products, giving details 
such as how well certain models work with pet 
hair (effort, first-hand experience, original 
content). There are many descriptions of 
participants’ own experiences with this product 
and how well it works for them, making the MC 
satisfying for the purpose of the page. 


| | | p 


Lowest Medium Highest 


This forum is a go-to source for discussions on 
luxury designer purses (positive website 
reputation, experience, expertise, "go-to" 
authority). On this particular forum page, 
members are consulting forum experts who 
have expertise authenticating bags from this 
brand. These experts can tell if a particular bag 
is authentic or fake. The quality of the MC is 
high due to the level of effort, skill, and 
engagement among those who participate in 
the forum. While there is an ad at the top and a 
few ads within the forum message, it does not 
distract from the MC, which is easy to find. 


| | | =a | 


Lowest Medium Highest 


This discussion focuses on the landscaping for 
a particular paludarium (an aquarium with 
terrestrial and aquatic elements). There isa 
lot of discussion and interaction between forum 
members about the types of materials and 
species used in the aquarium (effort, original or 
unique content). The posts show expertise in a 
niche topic aquarium landscaping (experience, 
expertise). 
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Webpage’/Type of Content 


Highest: Q&A page about how 
long most cancer patients live 


Highest: Q&A page on 
abbreviations 


Characteristics of the Page 


e Very satisfying MC: achieves the purpose 
of the page very well 

e Positive website reputation for the topic of 
the page 

e Very high E-E-A-T for the purpose of the 
page 


e Satisfying MC: achieves the purpose of 
the page well 

e Very positive website reputation for the 
topic of the page 

e Very high E-E-A-T for the purpose of the 
page 
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PQ Rating and Explanation 
| | | | 


Lowest Medium 


= 


Highest 


How long a person may live with a disease 
could significantly impact life decisions. This is 
a YMYL topic. 


This is a Q&A page on a medical question that 
is well-answered by non-experts with life 
experience. There are many responses 
describing how long a loved one lived after 
diagnosis. The focus of the page is 
experience, not medical advice. Many 
responses are based on personal experience 
and are helpful for this topic. 


Lowest Medium 


=j 


Highest 


The purpose of the page is to answer questions 
about the usage of abbreviations, and it 
contains helpful information about common 
abbreviation issues. This website has a very 
good reputation as a reference for information 
on writing, publishing, etc. It is considered 
highly authoritative and trustworthy for the topic 
of the page. 
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10.0 Page Quality Rating Tasks 
At first, PQ rating may seem difficult. There are several aspects of the page and the website to look at and think about. 


Important: Do not struggle with each PQ rating. Please give your best rating and move on. If you are having trouble 
deciding between two ratings, use the lower rating. If you are torn between three ratings, choose the one in the middle. 


Do not consider the country or location of the page or website for PQ rating. For example, English (US) raters should use 
the same PQ standards when rating pages from other English language websites (UK websites, Canadian websites, etc.) 
as they use when rating pages from U.S. websites. In other words, English (US) raters should not lower the PQ rating 
because the page location (UK, Canada) does not match the task location. 


These guidelines are specific to "regular" webpages. Occasionally, you may be asked to rate a landing page that is not a 
webpage. For example, you may be asked to rate a PDF file, a PNG or JPEG image file, etc. When the landing page of 
the URL is not a webpage, some of the criteria in these guidelines may not apply. In this case, please use your judgment. 


Finally, this Page Quality Rating Guideline does not completely cover every aspect of page quality. If you find pages that 
you truly believe to be High or Low quality, please rate them as such, even if the reason is based on something not 
covered in these guidelines. Please use the comment section to explain your reasoning. As always, use your judgment. 


10.1 Instructions for Rating Page Quality Tasks 
The Page Quality task page is broken up into several parts: 


1. Some initial questions about the task landing page. 

2. A “PQ grid” to record your observations about PQ characteristics of the landing page. 
3. The Overall PQ rating slider which records your Overall PQ rating. 

4. Acomment box to explain your rating. 


Some results to the initial questions will end the task early. If the page is Porn, Foreign Language, Did Not Load, or 
consists of restricted or inaccessible MC (e.g., subscription is required to view MC), you will not fill in the PQ grid or assign 
an overall rating. 


e Foreign Language should not be used when the language on the landing page is in the task language, a 
language that is commonly used by a significant percentage of the population in the task location, or English. 

e Did Not Load should be used for pages where there is absolutely no content on the page created by the website. 
There is no MC, SC, or Ads on the page. See this Wikipedia article for descriptions of different types of error 
messages. 


The PQ grid is designed to be your "note pad." It allows you to record your observations about the landing page and the 
website it belongs to. 
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11.0 Page Quality Rating FAQs 


Question 


Why do we have to do all 
these steps? This takes a 
long time! 


Are we just giving High 
quality ratings to pages that 
“look” good? 


Is sharing a personal 
experience or opinion a 
beneficial purpose? What if 
the personal opinion is 
upsetting or offensive? 


Can life experience justify a 
Highest quality rating for a 
page on a YMYL topic? 


You talked about expertise 
when rating MC. Does 
expertise matter for all 
topics? Aren't there some 
topics for which there are 
no experts? 


Answer 


With practice, the amount of time needed for accurate PQ ratings will decrease. The 
steps are important and are designed to help you assess many different aspects of PQ. 
You may be surprised by what you find. Pages that initially look Low quality may turn 
out to be Medium or High quality with careful inspection. The reverse may happen as 
well. We want your most informed, thoughtful opinion. 


No! The goal is to do the exact opposite. These steps are designed to help you 
analyze the page without using a superficial “does it look good?” approach. 


Sharing a personal experience or opinion can be a beneficial purpose as it allows 
people to appreciate different perspectives. Content on the Internet can help people 
understand why others think and feel the way they do. 


It can be valuable to see the opinions and perspectives of others, even when they are 
off-putting, distasteful, upsetting, or offensive to you personally. Apply the standards of 
these guidelines rather than your own personal opinions or views. 


However, sometimes opinions and perspectives can spread misleading information or 
otherwise have the potential to cause harm. Therefore, apply the standards in these 
guidelines to all types of pages and content. 


Remember: Your assessment of harm happens before and overrides other 
considerations such as the life experience or expertise of the content creator. 


Factual information and advice on YMYL topics should come from experts. Sharing life 
experience on YMYL topics can be Highest quality, but it must be trustworthy, safe, 
consistent with well-established expert consensus, and speak to topics that life 
experience is valuable for. 


For example, information about when and where to vote is a YMYL topic. It's possible 
to have a Highest quality page that shares life experience about voting, such as an 
inspiring personal post about becoming a citizen and voting for the first time. However, 
if a page about someone's voting experience gives inaccurate information about when 
and where to vote either deliberately or through careless oversight, the page should be 
rated Lowest because it could cause others to miss the opportunity to vote. 


Remember that we are not just talking about formal expertise. Informal expertise is 
equally important, and for some topics may be a more common type of E-E-A-T. 


For most page purposes and topics, you can find experts even when the field itself is 
niche or non-mainstream. For example, there are expert alternative medicine websites 
created by leading practitioners of acupuncture, herbal therapy, etc. There are also 
pages about alternative medicine written by people with no expertise or experience. 
E-E-A-T should differentiate between these two scenarios. 


Reminder: If the page is harmful to people or society, untrustworthy, or spammy as 


defined in these guidelines, expertise and experience doesn't matter. It should be rated 
Lowest. 
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Question 


When | think about E-E-A-T, 
it seems like expertise and 
experience overlap a lot. 
What's the difference? 


Aren't some types of pages 
always Low quality, such as 
celebrity gossip? 


I've never seen a High 
quality page of type X. If 
there are no high quality 
pages of this type, why are 
we giving existing pages a 
Low quality rating? 


Some of these criteria seem 
unfair. For example, some 
art pages do not have a 
purpose. Are these pages 
Low quality? 


How should interstitials 
factor into my rating? 


Answer 


Here's one way to think about it: 
e Expertise often involves objective, testable knowledge or skills, for example: 
can you calculate the load bearing weight of a bridge? 
e Experience can be more subjective and is often shared through personal 
narration, for example: how does it feel to experience love for another person? 


For PQ rating, it's not important to distinguish between expertise and experience. 
Instead, focus on what kind of content is trustworthy and satisfying for the purpose of 
the page. You may find that both expertise and experience can be trustworthy and 
satisfying for the same topic. For example, a medical website may be a trustworthy 
source for treatment options, and the experiences of other people who have gone 
through treatment may provide emotional support and prepare you for what to expect. 


For almost any type of page or informational topic, there is a range of content quality. 


For example, there are both High and Low quality celebrity gossip pages. The purpose 
of these pages is to share interesting, accurate information about celebrities. There 
may be benefits in celebrity stories that inspire others or help people going through 
similar challenges. A celebrity gossip page is High quality if it has accurate and 
interesting information, has no characteristics of Low or Lowest quality, and comes from 
a reasonably reliable source. 


For some topics or types of pages, there may not be many (or any!) High quality pages 
now, but there may be in the future. We need a uniform set of standards that apply to 
all pages, even for pages that have not yet been created. 


Art pages do have a purpose: artistic expression. Pages created for artistic expression 
do not deserve the Low quality rating simply because they have no other purpose. 
Artistic expression, humor, entertainment, sharing photos and videos, etc. are all valid 
and beneficial page purposes. 


Sometimes clicking on the task URL will bring up an interstitial page. You can ignore 
this page in your rating criteria if you can easily get to the MC. However, if the 
interstitial makes it extremely hard (or impossible) to get to the MC and evaluate how 
well the page achieves its purpose, that should factor into your PQ rating. 


Copyright 2023 82 


Part 2: Understanding Search User Needs 


12.0 Understanding Search Users, Queries, and Results 
Why do people search the Internet? 


People use Internet search engines to perform many different tasks in different environments using different types of 
devices, such as mobile phones, tablets, laptops, or computers. Keep in mind that searches can be simple or complex, 
and the underlying task that a person is trying to accomplish may take multiple steps to complete. For example, a simple 
task may be to find the director of a movie. A complex task may be to find a movie’s showtimes nearby, purchase tickets, 
and get directions to go to the theater. Overall, search engines should make it easy for people to complete tasks by 
surfacing helpful results right away. 


12.1 Important Rating Definitions and Ideas 


SERP for [iphone] 


Google 


= Should | buy an iPhone? 


Query: [iphone] 
Locale: English (US) 
User Location: New York, NY 


Query: This refers to the text that a user types or the audio a user speaks into a device. In these guidelines, queries have 
square brackets around them. If a user says “coffee shops near me,” we display: [coffee shops near me]. If a user types 
“flights to New York” in the search box, we display: [flights to New York]. 


There are many different types of queries because people use their devices to do many things, from searching for specific 
websites to looking for answers to specific questions to browsing videos for inspiration. 


User: The user is the person trying to find information or accomplish a task. Keep in mind that users are people from all 
over the world: people of all ages, genders, races, religions, political affiliations, etc. 


User Intent: When a person types or speaks a query, he or she is trying to accomplish something. We refer to this goal 
as the user intent. 


Locale: All queries have a locale, which is the language and region for the task. Regions are represented by a two-letter 
country code, for example, "US" in the locale "English (US)". For a current list of country codes, click here. We 
sometimes refer to the locale as the task location. 


User Location: Sometimes more specific information about the location of the person searching is provided, usually a 
city or state. In rating tasks, this information may be shown on a map. 


Search Engine Results Page (SERP): The page a search engine shows after a user enters a query in the search box. 
The SERP is made up of result blocks. 


Copyright 2023 83 


Result: We will use the word result to refer to the result block and the landing page. 


e Result Block: This is an individual “block” that appears on the user’s device in response to the query. The result 
block may display information in the block itself or contain links, or may do both. 


e The Landing Page (LP) is the page you see after you click a link in the result block. 


12.2 Understanding the Query 


Understanding the query is the first step in evaluating the task. Remember, a query is what a user types or speaks into 
their device. 


Some tasks include a query research link, which you should use if you don’t understand the query or user intent. 
Otherwise, please do web research using Google or an online dictionary or encyclopedia. If you still don’t understand the 
query or user intent, please release the task. 


Important: If you research the query on Google, please do not rely on the top results on the SERP. A query may have 
other meanings not represented on Google’s search results pages. Do not assign a high rating to a webpage just 
because it appears at the top of a list of search results on Google. 


Think about users in your locale typing or speaking the following queries into their phone. 


Query Likely User Intent 

[population of paris], English (US) Find the current population of Paris, France. 
[starbucks near me], English (US) Find the nearest Starbucks location. 

[weather], English (US) Find weather information in the user location right now. 


12.3 Locale and User Location 


All queries have a task language and task location (referred to in rating tasks as the "Locale"). The locale is important for 
understanding the query and user intent. Users in different locales may have different expectations for the same query. 


Query: [football] Query: [football] 
Locale: English (UK) Locale: English (US) 


Some tasks show a user location in addition to a locale. 
However, for many or most queries in a given locale, the user location does not change our understanding of the query 


and user intent. Here are some examples: [facebook.com], [pictures of kittens], [distance between the earth and the 
moon]. 
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When is the locale or user location important in understanding query interpretation and user intent? Please use both web 
research and your personal judgment to answer this question. Ask yourself, “Would people in one city or country be 
looking for something different than people in another city or country?” 


12.4 Queries with an Explicit Location 


Sometimes people tell search engines exactly what kinds of results they are looking for by adding the desired location in 
the query, regardless of their own location. We'll call this location inside the query the “explicit location.” The explicit 
location makes queries much easier to understand and interpret. 


Hotels | Dallas, TX d Hotels | New York, NY 
e SM © $239 |! © ¢ 3179 7 w >34 


` IJ $297 è € 
T Y CEST EANAN s DIAMOND A 
rammel! ” s (© $154 DISTRICT ș 


Crow) Park Fand Science k 
Va Grimes sa 2st LITTLE BRAZIL S 62 
DE sn? G SOUTH DALLAS QTimes Sq- 42 St Si oe 
arr ee N U & 
@ Dallas, Tx 22 @ New York, NY 22 
© Sun, Jun 12 Mon, Jun 13 2 “M . . 6 Sun, Jun 12 Mon, Jun 13 
ed mom IS comin i 
~y . g 2 ~ “| need a hotel for my 
Top-rated Budget options Luxury stays to visit. | need to find a Top-rated Budget options Luxury stays ` z 
j trip to New York 
Hilton Garden Inn Downtow... $108 hotel for her nearby RE DoubleTree by Hilton Hotel... $225 
4.3 te e ke kek (1.1K) 3.6 e k k + * (4.3K) 
GREATDEAL 26% less than usual £ = Free WiFi Query: [New York hotels] 
: S 
Query: [Dallas hotels] Locale: English (US) 
Hotel Crescent Court $239 Locale: English (US) The Westin New York at Tim... $179 ` : i 
4.5 kk kek H (1.3K) ae 4.3 e kek k h (4.1K) User Location: Dallas,TX 
DEAL 17% less than usual User Location: Dallas, TX GREATDEAL 32% less than usual nE ao 
_ i Explicit Location: New York 
Explicit Location: Dallas 
Omni Dallas Hotel $253 Î Pod 51 Hotel $163 
4.6 He ke k (7.7K) 4.0 He He k k * (2.8K) 
2. Outdoor pool = Free WiFi 
CANVAS Hotel | Dallas $113 The Empire Hotel $179 


t 
= 4.0 k + (3.5K) 


=> Free WiFi 


4.1 wk k k + (2.3K) 


œŒ Outdoor pool 


View 553 hotels > View 1,211 hotels > 


Sometimes the explicit location matches the user location or locale, and sometimes it doesn't. 


When there is an explicit location in the query, pay attention to it! People use explicit locations to indicate exactly what 
they are looking for. 


12.5 Queries with Multiple Meanings 


Many queries have more than one meaning. For example, the query [apple] might refer to the computer brand or the fruit. 
We will call these possible meanings query interpretations. 


Dominant Interpretation: The dominant interpretation of a query is what most users mean when they type the query. Not 
all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a 
little web research. 


Common Interpretation: A common interpretation of a query is what many or some users mean when they type a query. 
A query can have multiple common interpretations. 


Minor Interpretations: Sometimes you will find less common interpretations. These are interpretations that few users 
have in mind. We will call these minor interpretations. 
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Query: [apple] 
Locale: English (US) 


Computer company Fruit Person’s name City 
\ A A A © A ( : Apple, Oklahoma \ 
B S C : C } C = |) 
© © © © © © o A 
i i ki Hi i im! p) ig! ig! 
non ti il 
© © 
7 rT im m 7 im! igi 
nen 0 n 
$ ip! m i H 
NNNNA ui 
Dominant Interpretation Common Interpretation Minor Interpretations 


Query: [mercury] 
Locale: English (US) 


Planet Chemical element Car brand Insurance omp Boat engine company 


2 Hg $ 4 a T) = Pisce ‘mencuny 


\ INSURANCE GROUP j) The Water Calls 
C l } ( B Mercury } ( _ MERCU RY + 5 
© © © © © (1 @ 
ig! igi p 9' 1! i ig! 
E || i N ii il il 
® 
i”! ig im | 
non 
Common Interpretations Minor Interpretations 
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12.6 Query Meanings Can Change Over Time 


Remember to think about the query and its current meaning as you are rating. We will assume users are looking for 
current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., 
unless otherwise specified by the query. 


User in 1994 User in 2004 


Query: [George ; Query: [George 
Bush] l Bush] 
Locale: English (US) Locale: English (US) 


41* US president 


to 


43" US president 
to 


© (9 
Query: [iphone] Query: [iphone] 
Locale: English (US) Locale: English (US) 


iphone 13 


User in 2007 


iphone 1 


go Lu 


The interpretation of the query [iphone], English (US) has changed over time as new iPhone models are released. The 
first iPhone was introduced in 2007. Users searching for [iphone], English (US) at that time were looking for the new (at 
the time) first iPhone model. Most users now are looking for the most recent or upcoming iPhone model. In the future, 

new models will come out and the dominant interpretation will change again. 


12.7 Understanding User Intent 


It can be helpful to think of queries as having one or more of the following intents. 


Know query, some of which are Know Simple queries 

Do query, when the user is trying to accomplish a goal or engage in an activity 

Website query, when the user is looking for a specific website or webpage 

Visit-in-person query, some of which are looking for a specific business or organization, some of which are 
looking for a category of businesses 
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12.7.1 Know and Know Simple Queries 


The intent of a Know query is to find information on a topic. Users want to Know more about something. 


Know Simple queries are a special type of Know query. Know Simple queries seek a very specific answer, like a fact, 
diagram, etc. This answer has to be correct and complete, and can be displayed in a relatively small amount of space. 
As a general rule, if most people would agree on a correct answer, and it would fit in 1-2 sentences or a short list of items, 
the query can be called a Know Simple query. 


Know Simple queries may be questions such as [how tall is barack obama]. Frequently, Know Simple queries do not 
have question words. For example, [barack obama height] has the same user intent as [how tall is barack obama], but is 


not in a question format. 


Most queries are not Know Simple queries, such as: 


Broad, complex, and/or in-depth informational queries that do not have a short answer 
Ambiguous or unclear informational queries 
Informational queries on controversial topics 
Informational queries with no definitive “right answer” 
Queries where different users may want different types of information, or different sources of information 


Here are some examples where the Know Simple query asks for a simple fact, which can be answered correctly and 
completely in a small amount of space, and the Know query answer is more complex. 


Know Simple Query 


[barack obama height] 
[how tall is obama] 


[new york city population 2013] 


[who is graves disease named 
after] 


[macy’s store hours] 

[what is the symbol for the 
element nickel] 

[who won the 2014 bcs national 


championship game] 


[what is starbucks stock price] 


[weather] 


Know Query 


[barack obama] 


[new york city] 


[graves disease] 


[macy’s gift wrap options] 


[what nickel is used for] 


[who is going to win the bcs 
national championship game] 


[should i invest in starbucks stock] 


[how do people predict the 


weather] 
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Explanation 


The Know query is a broad information query and different 
users may be looking for different things (e.g., biography, 
books, social media posts, etc.). 


The Know query is a broad information query and different 
users may be looking for different things (e.g., tourist and trip 
planning information, facts, photographs, history). 


The Know query is a broad query for medical information 
and different users may have different needs. There is no 
single “answer” for this query. 


The Know query is a broad query for a particular service 
offered by a department store, and does not have a short 
answer. 


The Know query is a broad query and there is no short, 
complete answer. 


The Know query asks for an opinion and there is no 
definitive answer. 


Even though the Know query is theoretically a yes/no 
question, there is not a single answer that everyone would 
agree on. 


The query [weather] may seem like a broad information 
query, but most people likely have a fairly simple 
informational need: to find the current or upcoming 
temperature, and the chance of local weather events such as 
rain or snow. Therefore, consider queries like [weather], 
[weather today], [weather tomorrow], [weather this week] to 
be Know Simple queries. 


In contrast, the Know query is a broad information query and 
different users may be looking for different things. 
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12.7.2 Do Queries 


The intent of a Do query is to accomplish a goal or engage in an activity. The goal or activity may be to download, to buy, 
to obtain, to be entertained by, or to interact with a website or app. Users want to Do something. 


Here are some examples. 


Query Likely User Intent 

[get my GED online] Obtain U.S. high school equivalency diploma online. 
[online personality test] Take an online personality test. 

[what is my bmi?] Calculate BMI (body mass index). 

[pay parking ticket] Pay for a parking ticket. 

[watch stranger things] Watch the TV show “Stranger Things”. 


12.7.3 Website Queries 


The intent of a Website query is to locate a specific website or webpage that users have requested. This single webpage 
is called the target of the query. 


One type of Website query is a URL Query, which can be: 


e Exact, perfectly-formed, working URLs, such as [http:/Awww.ibm.com] or [www.ibm.com] or [ibm.com]. 
e Imperfect URL queries: Queries that look like URL queries, but are not “working URLs”. These URLs do not load 
if you type or paste them into your browser address bar. Even so, we believe users have a specific page in mind. 


Here are some examples. 


Query Likely User Intent 

[reddit login] View the login page for Reddit. 

[tiktok] View the TikTok website. 

[yahoo mail] View the Yahoo Mail website. 

[walmart.com] View the Walmart website. 

[featherchair.com featherweight 13.5 Ibs. View the Featherweight 13.5 lbs. wheelchair product page on the Feather Chair 
wheelchair] website. 
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12.7.4 Visit-in-Person Queries and User Location 


People use search engines to look for Visit-in-Person information, such as finding nearby coffee shops, gas stations, 
ATMs, restaurants, etc. 


Some queries clearly “ask” for nearby information or nearby results (e.g., businesses, organizations, other nearby places). 
Some queries are not asking for nearby information or nearby results. Here are some examples. 


Queries with Visit-in-Person Intent Query with Non-Visit-in-Person Intent 


Q car repair shop 


‘All Maps Images News Videos shoppin 


Auto Repair Shops 
near Oakland - Choose area 
2. how tall is mt everest x 


All Images News Books Videos Maps 


Mount Everest 


— tas Ê ~ “How tall is Mount Everest? 


29,032' 


2 © “lwant to find a car repair 
shop.” 


Query: [how tall is mt everest] 
Locale: English (US) 


Query: [car repair shop] bicu Evarest peat ta figs atuda above 
Loca le : E ng | is h (U S) att at 29,029 feet [8,848 meters]. 


& https/oceanservice.noaa.gov » facts H 
What is the highest point on Earth as 
measured from Earth's center? - 

35+ year: i i 

hosed "Opens BAM Tue National Ocean Service 


More businesses > 


, gas stations 
Q igtest 
Al Maps News images 
‘All Books News Images Videos Maps 


Gas Stations 


Boe netest.rg i 
eo oen zg w “| need to find a gas Take an instant IQ Test now. We are f ti A 
a g the most accurate IQ Test available ... 4 Wu | would like to take an IQ 


Oper ho 
i » m 
SN | st at ion rig ht away! Online 19 Test. Accurate Testing. Instant Results. Test 
= mag j ur ills wi x if ” 
A ues etn Rese & FAK Acar. test online. 


ay: ore Si n Oaia E 
tpn ees et 


— Query: [gas stations A S l 
ae lish as cal 5 I Query: [iq test] 
vas seein mart ont Locale: English (US) 


Certified Test 


About 


More examples: [pizza], [yoga class], [grocery stores], [movie More examples: [boston red sox], [scrabble cheat], [definition of 
showtimes], [car repair], [dentists], [bank of america atm sedentary], [aapl], [beyonce], [minecraft], [small dog breeds], 
locations], [library near me] [dance videos], [oscars], [silly jokes], [bank of america login] 
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And some queries could go either way. Some users may want nearby results and others may not. 


= Search everything a = 
$0.00 


x BARTON G@ How do you want your items? | 78735 7 
Barton Cree 
yy Ate Spoke Q Reserve pickup or delivery 
i EAST E HILL Ca 
pee at eee a a f “ f e H 
say Central Market $ 2 ~ “I want to go toa Major summer deals 4 ~ “I want to shop online 
Za AS Ssni Walmart near me.” i a on the Walmart website” 
$i WESTGATE 
Wholesale © jà Cathedral 
ei] Y 
a he Query: [walmart] Sa Query: [walmart] 
itk z A 
X es Rid Locale: English (US) Stop Uotoao%erttone Locale: English (US) 


SOUTH AUSTIN 


S The Little Dar 
y E 
James Bowie High School è | 


Up to 40% off toys 60% off select tech 


Here are some examples of queries with both visit-in-person and non-visit-in-person intent: 


[hotels] 

post office] 

[h&m] 

[library] 

[bank of america] 


[the gap] 
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Sometimes, the user location can change our understanding of the query. For users close to Austin, Texas, the query 
[verbena] could have two different interpretations: a popular restaurant named Verbena or the plant verbena. 


pa 
Verbena i Z A wawa Verbena 
Pedda ah 4 d From Wikipedia, the free encyclopedia 
= = ~ This article is about the plant genus. For other 
Mom RS hots Sages x uses, see Verbena (disambiguation). 
- "Vervain" redirects here. For other uses, see 
— > - 
(&) ( ©) (a) (© ) = Vervain SAEN 
CALL DIRECTIONS SAVE WEBSITE “ . zs Verbena (/vər'bi:nə/),!’] also 3 “ 
£ © “Whatis the address == Vereeratorbinoni® also o ~ “I want to learn more 
ead | Comal, of Verbena?” “== is a genus in the family about the plant verbena.” 
— Verbenaceae. It contains about 
Hip eatery serving oysters, charcuterie boards, steaks & fish, plus I 150 species of annual and 
weekend brunch, wine & beer. 1 
. o | herb: j- . 
Query: [verbena] See) Perens! herbaceous or semi Query: [verbena] 
{Œ} Dine-in- Takeout = woody flowering plants. The S 
», Locale: English (US) Z. majority of the species are native = Locale: English (US) 
Dova own : ‘ 4 = to the Americas and Asia; 4 A . 
O 612 W 6th St, Austin, TX 78701 agn User Location: Austin, TX LATER aula fie User Location: Austin, TX 
Austin common vervain or common 
© Closed- Opens SPM v verbena, is the type species, and 


Confirmed by this business 3 weeks ago native to Europe. 


X® Menu - verbenaatx.com 


\ (812) 991-3019 


In most other user locations, there is no restaurant (or anything else) named Verbena and there is just one interpretation 
of the query [verbena]: the plant. The Austin restaurant is not well-known outside of Austin, TX. 


wwa Verbena 


From Wikipedia, the free encyclopedia 
This article is about the plant genus. For other 
uses, see Verbena (disambiguation). 
"Vervain" redirects here. For other uses, see 
Vervain (disambiguation). 
Verbena (/var'bi:na/),!°! also 
known as vervain or verveine, 


2 > “| want to learn more about 
the plant verbena.” 


is a genus in the family 
Verbenaceae. It contains about 
150 species of annual and 

ə perennial herbaceous or semi- 


Query: [verbena] 
Locale: English (US) 
User Location: Flint, MI 


woody flowering plants. The 
majority of the species are native ° 
to the Americas and Asia; 
however, Verbena officinalis, the 
common vervain or common 
verbena, is the type species, and - 
native to Europe. 


Use your common sense when thinking about queries and whether they have possible visit-in-person intent. 
12.7.5 Queries with Multiple User Intents 


Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more 
likely than another intent. 


Query Likely User Intent 


Depending on the user need and location, users may want to visit the official homepage (Website), 


[harvard] get directions (Visit-in-Person), or learn more about the school (Know). 


Most users want to go to a nearby Walmart (Visit-in-Person) or view the homepage to shop online 


[walmart] (Website). Some or few users may want to learn more information about the company (Know). 
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12.8 Understanding Result Blocks 
The following sections contain examples of different types of queries and results. 
12.8.1 Web Search Result Block Examples 


Web Search Result Blocks typically have a title link, a URL and a “snippet” of text describing the page. For many queries, 
Web Search Result Blocks are the most helpful type of result. 


Query and User Intent Web Search Result Block 


GH] https://www.goodhousekeeping.com > ... 


9 Best Food Processors of 2022, 
According to Kitchen Appliance Experts - 
Good Housekeeping 


Query: [cuisinart food processor reviews] 
User Intent: This is a Know query. The user 


wants to find recent reviews of Cuisinart food 
processors. 


Result: This is a Web Search Result Block 
that has a link to a landing page with Cuisinart 
reviews. 


Dec 9, 2021 — Ranking at the top of our tests, this 3-cup 
Cuisinart is a great do-it-all size: The chop and grind 
functions provide versatility to mince ... 


Approx. $40 to $1,499 


E>  https://www.broadway.com 


Broadway Tickets | Broadway Shows | 
Query: [broadway tickets] Theater Tickets | Broadway.com 


User Intent: This is a Know query or Do The most comprehensive source for 
query. The user wants to search prices and/or Broadway Shows, Broadway Tickets, Off- 
purchase tickets to a Broadway show in New Broadway, London theater information, 


York City. 
ome Tickets, Gift Certificates, Videos, ... 


Shows Tickets The Broadway Show Company Ticke' 
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12.8.2 Special Content Result Block Examples 


Special Content Result Blocks (SCRBs) appear in the search results page, along with Web Search Result Blocks. 


Special Content Result Blocks are designed to show content directly to users on the search results page. From working 
calculators to playable videos to interactive weather information, these results help users immediately get information or 


content. Some SCRBs may also have links to landing pages. 


Important: Please assume that any interactive features work and function properly. Some notes: 


e All result blocks are “screenshots” or images of search results with prominent links enabled. Unfortunately, a 
screenshot or image of an interactive result block will not function as it would for a real user. For the purpose of 
rating, please assume that interactive result blocks do function as intended. Try to interact with the result 


block as some links, buttons, or other features may work. 


e There may be a delay between when the rating task is created and when you actually rate the block, causing 
some information in special content result blocks to be a few hours or even days out of date. Stock price or 
weather informational blocks are designed to give users extremely current and timely information. However, due 
to a delay in rating time, the information may no longer be accurate. Don’t penalize a special content result block 
for being out of date. Assume that the blocks show current information for users, unless instructed 


otherwise. 


Query and User Intent Special Content Result Block 


Chicago, IL 


PM 


Query: [weather] 92" ise 
User Location: Chicago, Illinois 


User Intent: This is a Know Simple query @ 
since users have a fairly simple informational M AM 
need: find the current temperature and chance 


of rain or snow. The user wants to know the FRI SAT SUN 
weather for the User Location. > > 
Note: Assume the block shows current 27° 20° 32° 
information for users. e s: se 
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26° 
17° 


Precip: 70 
Humidity: 73% 


Wind: 11 mph 


TUE wi 
35° 3 
32° 21 
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Query and User Intent 


Query: [emma stone movies] 


User Intent: This is a Know query. The user 
wants to get information on movies with Emma 
Stone. 


Result: In this result block, users can 
immediately see some popular movies starring 
Emma Stone, with an option to click on the 
links to learn more about each movie. Users 
can also swipe to see a list of more movies. 


Query: [calories in a banana] 


User Intent: This is a Know Simple query. 
The user wants to find out how many calories 
are in a banana. 


Special Content Result Block 


Popular movies starring Emma Stone 


FAVOURITE 


a 


2021 2018 


2017 


Romantic movies starring Emma Stone 


More movies v 


Banana / Energy Amount 


105 calories 


Bananas 


1 medium (7" to 7-7/8" long) (118 g: > 
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ASB ay 
CRAZY,STUPID,LOV 
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Query and User Intent Special Content Result Block 
NFL ig 
Pro Bowl 


Overview Games Standings News 


NFL Divisional Round 
Final/OT 
Pd Bills Sun, Jan 23 
G& Chiefs = & 
Query: [NFL scores] 


NFL Conference Championships 


User Intent: This is a Know query. The user Finel/OT 
wants to find the most recent National Football 1B Bengals Sun, Jan 30 


League scores. G& Chiefs 


NFL Conference Championships 
Final 
@ 4%ers Sun, Jan 30 


£A Rams 


More games v 


Fix You 
Song by Coldplay 


Overview Lyrics Listen Other recordings 


Query: [coldplay fix you video] 


User Intent: This is a Do query. The user 
wants to watch the music video for the song 
“Fix You” by Coldplay. 


Result: By clicking on this result block, users @ Coldplay - Fix You (Official Video) 


can play the music video for the song, learn Scio ae 


more about the artist/album, etc. 
Lyrics 
When you try your best, but you don't succeed 
When you get what you want, but not what you need 


When you feel so tired, but you can't sleep 
Stuck in reverse... 


Source: Musixmatch 


Fulllyrics > 
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Query and User Intent Special Content Result Block 


Tallest tree 


Query: [what is the tallest tree] 


User Intent: This is a Know Simple query. 
The user wants to know what type of tree is 
the tallest. 


The tallest trees in the world are redwoods 
(Sequoia sempervirens), which tower above the 
ground in California. These trees can easily 
reach heights of 300 feet (91 meters). Among 
the redwoods, a tree named Hyperion dwarfs 
them all. The tree was discovered in 2006, and 
is 379.7 feet (115.7 m) tall. Apr 17, 2013 


©  https://www.livescience.com > 2872... 


What Is the World's Tallest Tree - Live 
Science 
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Part 3: Needs Met Rating Guideline 


13.0 Rating Using the Needs Met Scale 


There are many different kinds of queries and results, but the process of rating is the same: Needs Met rating tasks ask 
you to focus on user needs and think about how helpful and satisfying the result is for the users. 


Needs Met Rating 


R | | | | | 


N/A FailsM SM MM HM FullyM 


This is what the Needs Met rating slider looks like: 


Porn: No. Foreign Language: No Did Not Load: No 


Rating Description 


A special rating category, which only applies to certain queries and results. All or almost all 
Fully Meets (FullyM) users would be immediately and fully satisfied by the result and would not need to view other 
results to satisfy their need. 


Highly Meets (HM) Very helpful for many or most users. Some users may wish to see additional results. 


Helpful for many users OR very helpful for some users. Some or many users may wish to 


Moderately Meets (MM) see additional results. 


Helpful for fewer users. There is a connection between the query and the result, but not a 


slightly Meets (SM) strong or satisfying connection. Many or most users would wish to see additional results. 


Completely fails to meet the needs of the users. All or almost all users would wish to see 


pails ta Maat (FailsM) additional results. 


Please note that you may assign in-between ratings. Use in-between ratings if you think the rating of a result falls 
between two labels. You can either drag the slider or click on the point that you want the slider to land on. 


13.1 Rating Result Blocks: Block Content and Landing Pages 


For Needs Met rating, you will assign a rating to each result. Each result includes the content inside the result block and 
landing pages associated with the result. 


Which part of the result do you rate? It depends on both the query and the result block: 


Type of Block What to Rate 


The content inside this type of block should always play a large role in your rating. 
Special Content Result Block 


(SCRB) Some Special Content Result Blocks may have links to landing pages. In these 
cases, think about whether a user would click on the link in order to satisfy their 

Note: Assume that interactive user need. 

result blocks function as intended. 

Try to interact with the result block e |f most users would not click, rate the Special Content Result Block based 

as some links, buttons, or other on the block content alone. 

features in your rating task may e |f some or many users would click, you may consider the helpfulness of the 

work. landing page(s) in addition to the content in the block. In this case, both 


need to be helpful to justify a high rating. 


Web Search Result Block A click is required, and you should evaluate the landing page to assign a rating. 
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For example, think about the query [what does love mean]. 


Result Block 


Special Content Result Block 


Web Search Result Block 


Hevi 4 
noun 
1. an intense feeling of deep affection. 
"babies fill parents with intense feelings of love" 
synonyms: deep affection, fondness, tenderness 


warmth, intimacy, attachment 
endearment; More 


. a person or thing that one loves. 
“she was the love of his life” 
synonyms: beloved, loved one, love of one’s life, 
dear, dearest, dear one, darling. 
sweetheart, sweet, angel, honey: More 


feel a deep romantic or sexual attachment to 

(someone). 

"do you love me?" 

synonyms: care very much for, feel deep affection 
for, hold very dear, adore, think the world 
of, be devoted to, dote on, idolize. 
worship; More 


Translations and more definitions 


What Is Love? We Spoke to the Merriam- 
Webster Lexicographer ... 
www.themarysue.com/definition-of-love/ 
Mobile-friendly - Nov 7, 2013 - “Love” is a word we all 
use and understand, but defining such a complex 
idea seems like a daunting ... 
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Most users would probably not click on 
the Special Content Result Block, 
because the block contains a large 
amount of helpful content and has no 
obvious landing page link associated 
with it. 


Therefore, base your rating on the 
content inside the block itself. 


In this case, users would have to click on 
the web search result in order to get an 
answer to the question. 


Therefore, base your rating on the 
content of the landing page. 
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Here are some examples of Special Content Result Blocks where the block should be rated primarily on the content inside 
the block itself. As always, please use your judgment. 


Query and Special Content Result Block 


Query: [utopia animal hospital] 


Court Ave 
ICAL 
RICT Court Ave 


Madison Ave 9 


Madison Ave 


Monroe 
Utopia Animal Hospital 
4.5 kk kK 18 reviews 


Animal Hospital 

1157 Madison Avenue, Memphis, TN 38104 
(901) 746-8758 

» Closing soon - 7:30 am — 5:30 pm 


N Y © 


Call Directions Website 


More about Utopia Animal Hospital 


Query: [how to find security code on 
visa] 


The card security code 

is located on the back of 
MasterCard, Visa and 

Discover credit or debit 

cards and is typically a separate 
group of 3 digits to the right of the 
signature strip. On American 
Express cards, the card security 
code is a printed, not embossed, 
group of four digits on the front 
towards the right. 


Card security code - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/.../Card_security_cod... 


Feedback 


Query: [chicago weather] 


Chicago, IL 


Fri, 5 PM, Snow showers 


Precip: 70% 
Humidity: 73% 
Wind: 11 mph 


22" 


35° 
32° 


weather.com Feedback 


Query: [how did Lincoln die] 


Assassination 


Abraham Lincoln, Cause of death 


“How Did Lincoln Die? Everyone knows that the 
ball John Wilkes Booth fired into Abraham 
Lincoln's brain inflicted a terrible, mortal wound. 
But when a prominent neurosurgeon began to 
investigate the assassination, he discovered 
persuasive evidence that Lincoln's doctors must 
share the blame with Booth's derringer.” 


How Did Lincoln Die? | American History Lives 
at American Heritage 
American Heritage » content » how-did-l... 


More about Abraham Lincoln 


Query: [calories in a banana] 


Banana / Energy Amount 


105 calories 


Bananas 


1 medium (7" to 7-7/8" long) (118 g + 


Query: [movies san francisco] 


Movies near San Francisco 


Genre 
All Genres ~v 


San Andreas Entourage 


v 


More movies 


Note: The guidance in this section specifically applies to Needs Met ratings. For Special Content Result Blocks that have 
landing pages, you may or may not also be asked to provide PQ ratings. In these cases, your PQ rating should always be 
based on the landing page. Please refer to Section 14.0 for more about the relationship between Needs Met and Page 


Quality ratings. 
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13.2 Fully Meets (FullyM) 
Fully Meets is a special rating category, which can be used in the following situations: 


e The query and user need must be specific, clear, and unambiguous. 
e All or almost all users would be completely satisfied by the result—users issuing that query would not need 
additional results to fully satisfy the user intent. 


In other words, the Fully Meets rating should be reserved for results that are the “complete and perfect response or 
answer’ so that no other results are necessary for all or almost all users to be fully satisfied. 


You will need to use your judgment to decide whether a result block can Fully Meet the user need. Here are some 
scenarios when the Fully Meets rating is appropriate: 


e The user is clearly looking for a specific webpage or website and the result block with the specific webpage or 
website Fully Meets the user’s need. 

e The user is looking for a very specific fact or piece of information and the result block provides the information 
thoroughly, accurately, and clearly, and is from a high quality source. No other results would be needed. Before 
using the Fully Meets rating for queries seeking a very specific fact or piece of information, you must check for 
accuracy and confirm that the information is supported by expert consensus where such consensus exists. 


Fully Meets may apply in other situations as well. Ask yourself whether the result block alone is the perfect and complete 
result that would fully satisfy all or almost all users. Be conservative when using the Fully Meets rating. When in doubt, 
consider a lower rating. 


Note: If a result block is very close to being fully satisfying, but the block alone may not be sufficient in order to fully satisfy 
all or almost all users, a rating of Highly Meets+ may be appropriate. 


Sometimes you have to think about how specific the user need is and will need to make a judgment call on whether it is 


specific enough to possibly have a Fully Meets rating. 


13.2.1 Examples of Fully Meets (FullyM) Result Blocks 


Query and User Intent Result Rating Fully Meets Explanation 
The query has clear intent to go 
Query: [amazon] Amazon.com to the amazon.com website. 
www.amazon.com . 
} i i i i While there may be other 
User Intent: Go to the tobile-friendly - Online retailer of books, movies Fak = eo iag FuM | interpretations for the query, the 
Amazon website music and games along with electronics, toys. dominant one is by far the 
` apparel, sports, tools, groceries and à 
website. 
Target : Expect More. Pay Less. 
www.target.com/ 
` Spend $50 and get free shipping on over 
Query: [target website] 500K items. Choose from a wide selection 
of furniture, kids ... i ' i l i The query has clear intent to go 
User Intent: Go to the Target — S = m Fuly | to target.com. 


, Mountain View Target Find Store 
website. 
Women's Electronics 


Home : Decor, Furn... Weekly Ad 
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Query: [shop nordstrom 
online] 


User Intent: Go to the 
Nordstrom website for online 
shopping. 

Query: [titanic imdb] 


User Intent: Go to the IMDb 
page for Titanic. 


Query: [www.yahoo.c0m] 


User Intent: Go to the Yahoo 
website. 


Query: [cnn health] 


User Intent: Go to the Health 
section of cnn.com. 


Query: [cnn] 


User Intent: Go to the CNN 
website. 


Query: [lebron james stats 
basketball- reference.com] 


User Intent: Go to the player 
statistics page for LeBron 
James on a specific website. 


Query: [chef chu phone 
number] 


User Location: Los Altos, 
California 


User Intent: Find the phone 
number for the restaurant 
called Chef Chu’s. 


Nordstrom 

m.shop.nordstrom.com/ 

Mobile-friendly - Shop online for shoes, clothing, 
jewelry, dresses, makeup and more from top 
brands. Make returns in store or by mail. 


Titanic - IMDb 
m.imdb.com/title/tt0120338/ 

kk k + Rating: 7.7/10 - 663,660 votes 
Mobile-friendly - Titanic -- Experience James 
Cameron's Titanic like never before. Leonardo 
DiCaprio and Kate Winslet Titanic -- Jack ... 


Yahoo 

https://www.yahoo.com/ 

Mobile-friendly - A new welcome to Yahoo. The 
new Yahoo experience makes it easier to discover 
the news and information that you ... 


Health News - Medical, 
Mental and Dental ... 


www.enn.com/HEAL TH/ 

Mobile-friendly - Discover the latest news and 
video about health at CNN. CNN covers diet, 
fitness , parenting, conditions and more, ... 


CNN.com 

www.enn.com/ 

Mobile-friendly - CNN.com delivers the latest 
breaking news and information on the latest top 
stories, weather, business, entertainment, ... 


LeBron James NBA & ABA Stats | 
Basketball-Reference.com 
www.basketball-reference.com > ... > J 

6/19 Factory of Sadness: Intel: LeBron 
James Opting in or Signing With Cavs: Just 
following the NBA Finals we wrote that 
Well, that's the latest talk, as noted by 
ESPN and others. 


San Antonio 
Shopping Center 


& (650) 948-2696 


Chef Chu's, Phone 


More about Chef Chu's 


j 
FailsM 


FailsM 
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| 
FullyM 


FullyM 


The query has clear intent to go 
to the Nordstrom website. 


The query has clear intent to go 
to the IMDb page for Titanic. 


Even though this is an imperfect 
URL query, it’s clear the user 
wants to go to the Yahoo 
website. 


The query has clear intent to go 
to the Health section of the 
cnn.com webpage. 


The query has clear intent to 
access CNN news and content. 
The website has the content the 
user is searching for, and fully 
satisfies the user intent. 


The query has clear intent to go 
to the player statistics page for 
LeBron James on a specific 
website, 
basketball-reference.com. 


Chef Chu’s is a Chinese 
restaurant located in the user 
location. This result block 
shows the correct phone 
number with the option to call 
the number directly on the 
device—it Fully Meets the user 
intent. 


Note: You must check for 
accuracy before using the Fully 
Meets rating. You can verify 
that the phone number in this 
SCRB is correct by checking 
Chef Chu's official website. 


102 


Query: [how to find Mac OS 
version] 


User Intent: Find out how to 
locate the OS version of a 
Mac. 


Query: [new york city 
population 2012] 


User Intent: Find the 
population of New York City 
back in 2012. 


Query: [what is the weather 
forecast for today] 


User Location: 
Mountain View, California 


User Intent: Find weather 
forecast information for the 
current day in Mountain View, 
California. 


Query: [what country is 
mount fuji in] 


User Intent: Find the country 
in which Mount Fuji is 
located. 


From the Apple menu é in the corner of your 
screen, choose About This Mac. You should 
see the macOS name, such as macOS 
Monterey or macOS Big Sur, followed by its 
version number. If you need to know the build 
number as well, click the version number to see 
it. 3 days ago 


@ https://support.apple.com > en-us : 
Find out which macOS your Mac is using - 
Apple Support 


8.337 million (2012) 


New York City, Population 


Los Angeles Chicago 
3.858 million (2012) 2.715 million (2012) 


Sources include: United Census Bureau 


Mountain View, CA 
Fri, 3 PM, Sunny 


FIC Precip: 0% 
= 68 Hum 40% 


Wind: 10 mph 


FRI 


> 


69° 
49° 


weather.com 


Japan 


Mount Fuji, Japanese Fuji-san, also spelled Fujisan, also 
called Fujiyama or Fuji no Yama, highest mountain in Japan. 
It rises to 12,388 feet (3,776 metres) near the Pacific Ocean 
coast in Yamanashi and Shizuoka ken (prefectures) of 
central Honshu, about 60 miles (100 km) west of the Tokyo- 
Yokohama metropolitan area. 


E https://www.britannica.com > place 
Mount Fuji | Facts, Height, Location, & 
Eruptions - Encyclopedia Britannica 


More about Japan > 


Copyright 2023 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users to 
read, and comes from a high 
quality, authoritative source for 
Apple product information. The 
result block Fully Meets the 
user need. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users to 
read—it Fully Meets the user 
need. 


The result block displays the 
complete and correct answer 
prominently in the result block in 
a way that is easy for users to 
read, and comes from a high 
quality source. This result Fully 
Meets the user need. 


Remember that we consider 
weather queries, like this one, to 
be a Know Simple query. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users to 
read, and comes from a high 
quality, trustworthy 
encyclopedia. The result block 
Fully Meets the user need. 
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Chevron 


Query: [chevron at shoreline 
and middlefield] 


User Location: Mountain 
View, California 


User Intent: Find the specific 
gas station specified by the 
user in order to visit the 
location in person. 


Query: [decemberists crane 
wife 3 lyrics] 


User Intent: See the lyrics to 
The Decemberists song 
called “Crane Wife 3.” 


Terra Bella Ave 


Cin 
/nda Vista Ave 


Chevron 
3.9 kkk 


Directions Share Website 


808 N Shoreline Blvd, Mountain View, CA 


94043 


More about Chevron 


The Crane Wife 3 


Song by The Decemberists 


Overview Listen Analysis 


And under the boughs unbowed 
All clothed in a snowy shroud 
She had no heart, so hardened 
All under the boughs unbowed 


Each feather it fell from skin 
‘Til threadbare, as she grew thin 
How were my eyes so blinded? 
Each feather it fell from skin 


And | will hang my head, hang my head low 
And | will hang my head, hang my head low 


A gray sky, a bitter sting 
Arain cloud, a crane on wing 
All out beyond horizon 

A gray sky, a bitter sting 


And | will hang my head, hang my head low 
And | will hang my head, hang my head low 
And I will hang my head, hang my head low 
And I will hang my head, hang my head low 
And I will hang my head, hang my head low 
And I will hang my head, hang my head low, low, low 


Source: LyricFind 
Songwriters: Colin Meloy 


The Crane Wife 3 lyrics © BMG Rights Management 
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The intent is to visit the specific 
gas station specified by the 
user. This result block is for the 
specified Chevron location, with 
information to visit this location 
in person. 


The result has exactly what the 
user wants, displaying the lyrics 
to the specified song clearly and 
completely inside the result 
block, and comes from a source 
with high E-E-A-T for this topic. 
The user does not need to click 
to a new page to see the 
complete lyrics for this song. 


Note: The lyrics posted on 
LyricFind are licensed. 


104 


Query and User Intent 


Result 


Query: [nearby coffee shops] 


User Location: Mountain 
View, California 


User Intent: Find coffee 
shops near the user location. 


Query: [1600 pennsylvania 
ave washington dc] 


User Location: Bakersville, 
California 


User Intent: Find a map, 
directions, information about 
what is located at this 
address, etc. 


Red Rock Coffee 
4.1 kkk kk (641 


Coffee Shop - Castr 


Starbucks 
3.8 kkk 


Coffee Shop - Pear 


Clocktower Coffee Roasting 
3.9 kkk 


j Rd 


More coffee shops 


H SUNW. H St NW. 


Lafayette 


The White House } 


_ W Washington D.C 


Yt Old Ebbitt Grill 
F St NW. 


MNAS GL 


MNAS USEL 


E StNW. 


1600 Pennsylvania Ave 
NW, Washington, DC 
20500 


| = 
Map data ih 


Rating 


b 
FailsM 
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| 
HM 


Fully Meets Explanation 


FullyM 


The intent is to find coffee shops 
near the specific user location, 
represented by the blue dot on 
the map. 


This result block has a very 
satisfying list of nearby, popular, 
and prominent options within a 
few miles of the specific 
location. This kind of block is 
especially helpful for users who 
want to visit the business in 
person. Note that these blocks 
are interactive—clicking on a 
coffee shop will give options to 
call the business, get directions, 
read reviews, etc. 


Note: A range is included 
because some users would be 
fully satisfied with these 
selections, while some users 
would want more information 
(e.g., hours, distance from 
location, longer list of options). 


This block contains a map with 
the queried address, link to get 
directions, and specifies the 
landmark at the well-known 
address on the map (“The White 
House”). 


It's not unusual to search for an 
address (or business) that is far 
from the user location. Here, 
the user has explicitly asked for 
a specific address in a far away 
place, and this result is very 
satisfying. For a query this 
specific, the user location does 
not change the rating. 


Note: A range is included 
because some users would be 
fully satisfied with this result, 
while some users would want 
more information (e.g., photos, a 
brief description). 
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Query: [metro center station] 


User Location: Adams 
Morgan, Washington D.C. 


User Intent: The user wants 
to see a map, schedule, 
transfer options, and/or other 
train information for the Metro 
Center station, given that the 
user is close to the queried 
station. 


yo SH St NW. 
aw 


yet 
L Metro Salt 


Old Ebbitt Grill 


MNS MOL 


DIRECTIONS 


Departures 


: i 
Shady Grove HM 


11:19AM > 
[Rea] 


Silver Spring 


Wiehle Reston East 


Departure board 


More about Metro Center station 
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Fully 


This block contains a map, link 
to get directions, departure 

times for the multiple train lines 
that operate at this station, link 
to a detailed schedule (i.e., the 
“Departure board”), and an 

option to see more information. 


Since the user is located in the 
same city as the station, the 
information in this block is highly 
satisfying. 


Note: A range is included 
because some users would be 
fully satisfied with this result, 
while some users may want 
more information (e.g., nearby 
attractions, ticket prices). 
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13.2.2 Examples of Queries that Cannot Have Fully Meets Results 


There are some queries that cannot have a Fully Meets result. Here are some examples. 


Type of Query Example = No Fully Meets Result: Explanation 


This is a broad informational query. Knitting 
is an activity anyone can do and that anyone 
can create a website for. Different users 
may want different types of content: videos, 
[knitting] instructions, patterns, etc. There is no one 
Broad queries where no official source for knitting information and no 
single result could fully one result could satisfy most users. 
satisfy all users Therefore, no Fully Meets result is possible 
for this query. 


Famous names This is a broad informational query and it is 
impossible to know exactly what the user is 
e.g., [barack obama] looking for. 


There is no dominant interpretation for this 
query. The following entities are all common 
interpretations: Americans with Disabilities 
Act, American Dental Association, and 
American Diabetes Association. While each 
interpretation has an official homepage, 
none is Fully Meets since there is no 
dominant interpretation. 


Ambiguous queries 
without a clear user intent 
or dominant interpretation 


Queries for people’s names can be tricky. 
Many or most people queries do not have a 
dominant interpretation. Even unusual 
sounding name queries may not have a 

Non-famous people names dominant interpretation. For example, the 
queries [sam wen], [tran nguyen], and [david 

e.g., [sam wen] mease] can have no Fully Meets result 
because there are multiple people with each 
of these names, and it is not clear that most 
users are looking for any one particular 
individual. 


Copyright 2023 107 


13.3 Highly Meets (HM) 


A rating of Highly Meets is assigned to results that meet the needs of many or most users. Highly Meets results are 
highly satisfying and a good “fit” for the query. In addition, they often have some or all of the following characteristics: high 


quality, authoritative, entertaining, and/or recent (e.g., breaking news on a topic). 


In order to receive a Highly Meets rating, information pages such as encyclopedia articles and news articles must be 
accurate and highly credible. Highly Meets medical and scientific information pages must represent well-established 
scientific/medical consensus unless the user is clearly seeking an alternative viewpoint. For all information pages, you 
must check for accuracy and confirm that the information is supported by expert consensus where such consensus exists. 


A query can have many Highly Meets results. 


Have high standards for the Highly Meets rating. This is especially important for queries with many on-topic results. 


13.3.1 Examples of Highly Meets (HM) Result Blocks 


Query and User Intent | Result Rating 


Shelby Gastonia $ 


Query: [trader joes] 


User Location: Charlotte, A Trader Joe's 
North Carolina ae \ a 


User Intent: There are i Tairi — 
two possible user intents: EA 
most users probably want 
to visit a nearby location or 
go to the website. 


C Trader Joe's 


Query: [trader joes] 


User Location: Charlotte, 


North Carolina AOA 


www.traderjoes.com 
' 


User Intent: There are bile-fr - Trader Joe's is a neighborhood FailsM 


grocery store with amazing food and drink from 
around the globe and around the corner 


two possible user intents: 
most users probably want 
to visit a nearby location or 
go to the website. 


Query: [museum of 
modern art] 


User Location: 


Manhattan, New York 
MoMA | Museum of Modern Art 


Museum of Modern Art > 
User Intent: There are Mobile-friendly - The Museum of Modern Art l 


two possible user intents: (MoMA) is a place that fuels creativity, ignites FailsM 
most Heels probably want ea pppn CiS. Buy Tickets 

to visit the museum (note 

that the user is located in 

NYC where the museum is 

located) or go to the 

website. 
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Highly Meets Explanation 


This result shows a complete list 
of all three locations in the 
Charlotte area, with information 
that is especially helpful for users 
who want to visit the store. 


Note: This result block is not 
Fully Meets because users who 
want to go to the website to see 
coupons, promotions, etc. would 
have to see additional results. 


This is the official website for 
Trader Joe’s, which has highly 
authoritative information on the 
businesses’ stores, promotions, 
recipes, news, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the store 
have to do a bit of work to find 
that information—it is not 
immediately available. 


This is the museum’s official 
website and has very helpful 
information about exhibits, 
membership, the store, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the 
museum have to do a bit of work 
to find that information—it is not 
immediately available. 
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Query: [museum of 
modern art] 


User Location: 
Manhattan, New York 


User Intent: There are 
two possible user intents: 
most users probably want 
to visit the museum (note 
that the user is located in 
NYC where the museum is 
located) or go to the 
website. 


Query: [prestige bmw] 


User Location: 
Ramsey, New Jersey 


User Intent: There are 


two possible user intents: 
most users probably want 
to visit the car dealership 

or go to the website. 


Query: [prestige bmw] 


User Location: 
Ramsey, New Jersey 


User Intent: There are 
two possible user intents: 
most users probably want 
to visit the car dealership 
or go to the website. 


Museum of Modern At + 
Museum in New York City, New York 
4.5 kk kkk 971 reviews 


U k © 


CALL DIRECTIONS WEBSITE 
9 11 W 53rd St, New York, NY 10019 
© Closing soon 10:30 am - 5:30 pm v 
Upcoming events 


Evening Tours of MoMA's Mast. 
Mother's Day at MoMA 


Profiles 


Oo B 


Facebook Twitter Google+ 


v Artwork and overview 


Prestige BMW 
4.4 khkk 138 reviews 
BMW Dealer 


© © 


DIRECTIONS WEBSITE 


fe) 985 NJ-17, Ramsey, NJ 07446 


© Closing soon 9:00 am — 8:00 pm v 


More about Prestige BMW 


Prestige BMW 
www.prestigebmw.com 

} 
Mobile-friendly - Prestige BMW is the new, used FailsM FullyM 
& CPO BMW dealership in Ramsey, NJ near 


New York. We serve the area's BMW lease. .. 
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The result block shows 
information about the museum, 
and is especially helpful for users 
who want to visit the museum. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


The query is the name of a car 
dealership in New Jersey, which 
is the only dealership with that 
name in the user location 
(Ramsey, New Jersey). The 
result block shows information 
about the dealership, and is 
especially helpful for users who 
want to visit the business. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


This is the car dealership’s 
official website and has very 
helpful information about 
inventory, financing, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the 
dealership have to do a bit of 
work to find that information—it is 
not immediately available. 


109 


Query: [belmont library] 


User Location: Belmont, 
Massachusetts 


User Intent: Find 
information about this 
library or go to the 
website. 


Query: [belmont library] 


User Location: Belmont, 
California 


User Intent: Find 
information about this 
library or go to the 
website. 


Query: [decemberists] 


User Intent: Learn more 
about the band, The 
Decemberists. 


Query: [celine dion] 


User Intent: Learn more 
about the singer Celine 
Dion. 


Channing Rd 
E Belmont 


"E N: 
apc 
Belmont Public Library 


Library in Belmont, Massachusetts 


2 reviews 


U 


CALL DIRECTIONS WEBSITE 


Q 336 Concord Ave, Belmont, MA 02478 


© Closing soon 9:00 am — 9:00 pm v 


tion: Library 


v 


More about Belmont Public Library 


Belmont Library 
4.5 kkk kk 10 reviews 
Library 


© © 


DIRECTIONS WEBSITE 


1110 Alameda de las Pulgas, Belmont, 
CA 94002 


© Open today 10:00 am — 9:00 pm v 


More about Belmont Library 


The Decemberists 

www.decemberists.com/ 

Mobile-friendly - New album - What A Terrible | 
World, What A Beautiful World - coming January FailsM FullyM 
20, 2015. Pre-order now. 

On-Tour - Discography - “V is for Victory” Benefit 


Celine Dion | The Official Celine Dion 
Site 

www.celinedion.com/ 

Celine Dion's official website featuring Celine news, 
music, videos, album info and more! 

Videos - What's Goin' - TeamCeline account 
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There is only one library in the 
user location (Belmont, 
Massachusetts). The result 
block shows information about 
the library, and is especially 
helpful for users who want to visit 
the location. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


There is only one library in the 
user location (Belmont, 
California). The result block 
shows information about the 
library, and is especially helpful 
for users who want to visit the 
location. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


The user intent is too broad to 
have a Fully Meets result. 
However, this is The 
Decemberists’ official website 
and has a lot of content including 
tour dates, music videos, the 
latest album, etc. The result is 
uniquely authoritative and most 
users would be very satisfied by 
this website. 


The user intent is too broad to 
have a Fully Meets result. 
However, this is Celine Dion's 
official website and has a lot of 
content including recent news, 
show information, music, photos, 
videos, etc. The result is uniquely 
authoritative and most users 
would be very satisfied by this 
website. 
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Query: [fear the walking 
dead] 


User Intent: 

Find information about the 
TV show (e.g., show 
summary, episode guide, 
cast information, 
streaming options, etc.). 


Query: [manresa reviews] 


User Location: San Jose, 
California 


User Intent: Find reviews 
for this restaurant located 
in Los Gatos, California. 


Query: [what country is 
mount fuji in] 


ing Dead 
eve 


8/10 15% 
IGN.com Rotten Tomat. 


There's a sign of the apocalypse in Los Angeles, 
where reports of a strange flu and the 
disappearance of people underscore this gritty 
drama, a prequel to AMC's uber-popular 
"Walking Dead." The story of the mysterious 
outbreak and its ramifications are told through 
the lens of high school guidance c... More 


Network: AMC 


Episodes 


10. Do Not Disturb 


Season 2: Sep 4, 2016 


9. Los Muertos 
Season 2: Aug 28, 2016 


8. Grotesque 


Season 2 - Aug 21, 2016 


> More episodes 


Profiles 


Facebook Twitter Instagram 


Alycia Debnam Cliff Curtis Frank Dillane 
Carey Travis Manawa Nick Clark 
Alicia Clark 


More about Fear the Walking Dead 


Manresa, Los Gatos - Menu, Prices & 
Reviews - TripAdvisor 
www.tripadvisor.com/Restaurant_Review... 
Akk * Rating: 4 - 127 reviews 
Mobile-friendly - Manresa, Los Gatos: See 127 
unbiased reviews of Manresa, rated 4 of 5 on 
TripAdvisor and ranked #12 of 139 ... 


Wikipedia 
https://en.m.wikipedia.org > wiki 


The user intent is too broad to 
have a Fully Meets result, and 
different users may be looking for 
different things. 


This interactive block has a large 
amount of helpful content for a 
TV show including a summary, 
episode and cast information, 
etc.—this result would be very 
helpful for many or most users. 


The LP for this web result has 
127 reviews for the restaurant. 
The first three reviews show 
automatically, and then you can 
click to see more. This result 
would be very helpful for many or 
most users. 


This is a specific Know Simple 
query. The LP for this web result 
provides the answer to the query 


Mount Fuji - Wikipedia | dl along with a lot of information 
User Intent: Find the Mount Fuji ($+, Fujisan, Japanese: [w+ (d)zisan] ( listen)), FailsM s FullyM about the mountain. While the 
a 7 or Fugaku, located on the island of Honshi, is the highest H 
country in which Mount eini E aS answer can also be found in the 


Fuji is located. description of the web result, it is 
not displayed prominently. 
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Query: [broadway tickets] 


User Location: New York, 
New York 


User Intent: Research 
prices and/or purchase 
tickets to a Broadway 
show. 


Query: [kids backpacks] 


User Intent: Research 
prices and/or purchase 
backpacks for kids. 


Query: [poison ivy] 


User Intent: Find pictures 
of poison ivy plants, 
information about how to 
treat poison ivy, etc. 


Query: [michael jordan] 


User Intent: Find 
information, news, images, 
etc. about Michael Jordan. 


Query: [dance video] 


User Intent: Find a dance 
video to watch. 


Broadway tickets, show dates, 
calendar. Official Ticketmaster site. 
www.ticketmaster.com/broadway 

Find and buy broadway show tickets at 
Ticketmaster.com, 


Kids’ School Backpacks & Messenger 
Bags | Free Shipping from L.L. ... 
www.IIbean.com/Ilb/.../818?...backpacks 

Our durable kids’ backpacks are loaded with 
pockets and features - like padded shoulder straps, 
nighttime reflective 


Outsmarting Poison Ivy and Other 
Poisonous Plants 

www. fda.gov/.../ucm049342,htm 

Sep 2, 2008 - Poison ivy and other poisonous plants 
are a hazard all year round. Here are tips for 
preventing and ... 


Michael Jordan - Wikipedia, the free 
encyclopedia 
en.wikipedia.org/wiki/Michael_Jordan 

Michael Jeffrey Jordan (born February 17, 1963), 
also known by his initials, MJ, is an American 
former professional ... 


Evolution of Dance - YouTube 
m.youtube.com/watc... 
Apr 6, 2006 
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I 
FailsM 


FailsM 


} 
FailsM 


j 
FailsM 


The LP for this web result offers 
many Broadway tickets for sale, 
from a highly trustworthy source. 
While you have to zoom in and 
scroll to navigate the site, this 
result would be very helpful for 
many or most users because it is 
so highly trustworthy. 


The LP for this web result offers 
many kids backpacks for sale. 
The company is well-known for 
manufacturing and selling high 
quality backpacks for kids of 
different ages. There is a lot of 
product information and many 
user reviews for each backpack, 
in addition to helpful filter 
options. This result would be 
very helpful for many or most 
users. 


The LP for this web result is part 
of an authoritative government 
website and provides a lot of 
helpful information about poison 
ivy. This result would be very 
helpful for many or most users. A 
few or some users may wish to 
see additional results. 


The LP of this web result is a 
Wikipedia article about Michael 
Jordan. This result would be 
helpful for many or most users. 


Users are looking for good or 
entertaining dance videos and 
there are many options online. 


There are many on-topic results 
for this query and this is one very 
popular video of a comedian 
demonstrating dance styles from 
previous decades. 
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Query: [trestle bridge] 


User Intent: The user 
wants to understand what 
a trestle bridge is or learn 
more information about 
them (e.g., what it looks 
like, how they are built, 
types of materials, 
well-known examples, 
etc.). 


Query: [purple 
coneflowers] 


User Intent: Find images 
of or information about 
purple coneflowers. 


Query: [invasive species] 


User Intent: Find more 
information about invasive 
species. 


Query: [verbena] 
User Location: Austin, TX 


User Intent: Find 
information about the 
spice or the restaurant in 
Austin. 


View all images 


Images 


More images for purple coneflowers 


An invasive species is a 

plant or animal that is not 

native to a specific 

location (an Introduced www nature org 
species); and has a tendency to 
spread, which is believed to cause 
damage to the environment, human 
economy and/or human health. 


Invasive species - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/wiki/Invasive_species 


Verbena iF 
4.2 tee t k + (175) < 
Restaurant - Closed 


Updates 


© © © © 


CALL DIRECTIONS SAVE WEBSITE 


RESERVE A TABLE ORDER ONLINE 


Hip eatery serving oysters, charcuterie boards, steaks & fish, plus 


weekend brunch, wine & beer. ? 


@) Dine-in- Takeout 
Sth sy 


i 
9 612 W 6th St, Austin, TX 78701 sapei 


Austin 


Closed - Opens 5PM v 
Confirmed by this business 3 weeks ago 


Menu - verbenaatx.com 


(512) 991-3019 
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Seeing images of trestle bridges 
is very helpful in understanding 
this type of bridge. In this case, 
“a picture is worth a thousand 
words”, meaning that a picture 
may be more helpful than a text 
description due to the unique 
design of the bridge. 


This block contains many helpful 
images, and the landing page 
shows even more images—it 
Highly Meets the user intent. 


The result block has a good 
summary of what invasive 
species are, but some users 
would probably want to see more 
information like examples, 
impact, etc. 


For users in Austin, TX, the 
query [verbena] could have two 
different interpretations: a 
popular restaurant named 
Verbena or the plant verbena. 
Because the user is located in 
Austin, many users are probably 
looking for the restaurant. 
However, it is not completely 
clear because some users may 
want to find information about the 
plant. This result about the 
restaurant has satisfying 
information for users who want to 
visit it—the result Highly Meets 
the most likely user need. 
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Query and User Intent 


Query: [how to get from 
seattle to xpppzyfii] 


User Location: Seattle, 
Washington 


User Intent: Get 
directions from Seattle to 
another place. Note that 
the location in the query 
(“xpppzyfii”) is garbled and 
itis impossible to know 
what the user wanted. 
(The name was likely 
mistyped or mistranscribed 
from a voice query.) 


Query: [kristen wiig] 


User Intent: Find more 
information about the 
actress, comedian, writer, 
and producer. 


Query: [greek alphabet] 


User Intent: Find 
historical information about 
the Greek alphabet, the 
names of the letters in the 
Greek alphabet, images of 
the letters, etc. 


Query: [jennifer aniston] 


User Intent: Find images, 
recent news, gossip, 
information, etc. about 
Jennifer Aniston. 


Result 


No results for that place. Try entering it 
below to get suggestions. 


Seattle, WA h 
FailsM 


Kristen Wiig | www.kristenwiig.org ° 
Your fan site for Oscar ... 
www.kristenwiig.org 

- Watch the full interview on the 
official website to hear Kristen and director 
Shira Piven talk about their careers in acting 
Photo Gallery - KristenWiig.org Video Archive 


j 
FailsM 


Mobile-frier 


Images 


ap yoe 

CHOvK : i 

Ruveo i m . ah } 
HOT i AOANE FailsM 
dO xipw 


More images for Greek alphabet 


In the news 


What Oscar Snubs? Inside Jennifer 
Aniston and Angelina Jolie's Night at 
the Critics' Choice Awards : 
People.com 


People.com - 5 hours ago 


EXCLUSIVE: Jennifer Aniston double 
heartbreak as BOTH her ‘lost love’ and best 
friend, who starred in cult classic ... 


FailsM 
Daily Mail - 2 hours ago 

Jennifer Aniston Reacts to Oscars 2015 

Cake Snub, Says Reese Witherspoon and 


Julianne Moore Emailed Her 


Us Magazine - 4 hours ago 


More news for jennifer aniston 
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Rating 


HM 


HM 


HM 


HM 


| 
FullyM 


I 
FullyM 


I 
FullyM 


| 
FullyM 


Highly Meets Explanation 


Even though the user has to 
re-enter the destination, the 
result block recognizes that the 
user was asking for directions 
from Seattle. The user has to 
enter the destination manually 
because the name in the query 
(“xpppzyfii’) is so garbled that 
there is no good guess as to 
what the user meant. 


Therefore, this block is satisfying 
because it helps the user correct 
the request and then immediately 
get the desired directions. 


This is a fan site dedicated to 
Kristen Wiig. The website has 
comprehensive info including 
over 50,000 pictures and 300 
video clips, interviews, articles, 
and more. This website Highly 
Meets the need of many or most 
users. 


This is a broad query. Different 
users may be looking for different 
things. 


This result shows multiple tables 
of images with the letters in the 
Greek alphabet. This result 
would be helpful for many users. 
Some users may wish to see 
additional results. 


This is a broad query for an 
actress. Different users may be 
looking for different things. 


This result shows recent news 
about Jennifer Aniston. The 
articles are timely (at the time 
this was written) and relatively 
interesting, and not just common 
everyday news about the 
actress. Many users would be 
satisfied with the result and some 
users may wish to see more 
results. 
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Query: [seattle, 
washington] 


User Intent: Find 


information, news, maps, 


etc. related to Seattle, 
Washington. 


Query: [emily blunt] 


User Intent: Find 
biographical information 
(e.g., filmography) or 
current news/celebrity 


gossip about Emily Blunt. 


Query: [coldplay yellow 
lyrics] 


User Intent: Find the 
lyrics to the song 
“Yellow”, which was 
written by the band 
Coldplay. 


Seattle 
City in Washington 


Weather: 52°F (11°C), Wind 0 mph (0 km 


Points of interest e= rom | 


Fully 


Pike Place Space Needle Experience 
Market Music Project 
Museum 


Upcoming events and overview 


Emily Blunt 
Actress 


OVERVIEW MOVIES 


Emily Olivia Leah Blunt is an English actress. She 
made her professional debut in The Royal Family. 
Her screen debut was in the 2003 television film 
Boudica. Wikipedia 


Spouse: John Krasinski (m. 2010) m=i m | 
FullyM 


Movies 


Edge of Sicario The Huntsmi 
Tomorrow 2015 Winter's War 
2014 2016 


More about Emily Blunt 


Coldplay: Lyrics: Yellow 
Coldplay > recordings > parachutes > yell.. 


The official Coldplay website, featuring news, lyrics, tour dates, 
blogs, interviews, pictures and videos. ... Yellow Look at the stars 
Look how they shine for you And everything you do Yeah, they 
were all yellow | came ... 
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This is a broad query for a city. 
Different users may be looking 
for different things. 


The result contains a wide 
variety of helpful information 
such as a map, weather 
information, several points of 
interest, and has an option to see 
more information containing 
nearby events and facts about 
the city. Moderately Meets+ to 
Highly Meets is an appropriate 
rating for this result. 


This is a broad query for an 
actress. Different users may be 
looking for different things. 


The result contains a variety of 
helpful information such as 
images, brief biography, recent 
movies, and an option to see 
more. Moderately Meets+ to 
Highly Meets is an appropriate 
rating for this result. 


This page provides the 
requested lyrics on Coldplay's 
official website. Users can 
also navigate around the site 
to listen to the song, explore 
all albums the song appears 
on, listen to or view lyrics for 
other songs on the same 
album, or learn more about 
the artist overall. 


Additionally, users can trust 
that the lyrics on the LP are 
100% accurate due to the 
uniquely authoritative nature 
of the source, which makes 
this result very helpful for 
many or most users. 


115 


Query and User Intent 


Query: [1969 moon 
landing] 


User Intent: Find accurate 
and authoritative 
information about this 
historical event. 


Query: [stem education] 


User Intent: Find 
information or recent news 
about education in STEM 
fields (i.e., Science, 
Technology, Engineering, 
and Mathematics). 


Query: [mentos and coke 
death] 


User Intent: Find accurate 
information about a rumor 
that a mixture of Mentos 
and Coca-Cola can cause 
death, which was spread 
on the Internet in 2006. 


Query: [pj fleck] 


User Location: 
Minneapolis, Minnesota 


User Intent: Find 
information or recent news 
about P.J. Fleck, who had 
just become the new head 
football coach at the 
University of Minnesota a 
few days before this 
example was written. 


Result 


July 20, 1969: One Giant Leap For Mankind | NASA 
NASA (.gov) > apollo > apollo11 


At 10:56 p.m. EDT Neil Armstrong is 
ready to plant the first human foot on 
another world. With more than half a 
billion people watching on television, he 
climbs down the ladder and proclaims: 
"That's one small step for a .. 


= 


Educators, advocates see increased importance 
in US STEM... 
Fox News > 2015/06/29 > educators-adv... 


AMP - Jun 29, 2015 - When it comes to education 
reform in the United States, Science, Technology, 
Engineering, and... 


FailsM 


Mentos and Coke Death : snopes.com 
Snopes > Fact Check» Horrors 

j 
Nov 9, 2006 - Mixture of Mentos and Coca-Cola kills child? FailsM 
„ Last week a little boy died in Brazil after eating MENTOS 


and drinking COCA COLA together. 


Welcome aboard, ‘life coach’ PJ. Fleck - 

StarTribune.com 

Star Tribune » welcome-aboard-life-coac... j 
FailsM 

Jan 9, 2017 - PJ. Fleck's tenure as head football coach 

at the University of Minnesota will not be defined by an... 
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Rating 


HM 


FullyM 


HM 


| 
FullyM 


| 
FullyM 


I 
FullyM 


Highly Meets Explanation 


The LP for this web result 
contains authoritative information 
about the Apollo 11 moon landing 
on the official website of NASA, 
which was the organization that 
oversaw this mission in 1969. 
The MC has a lot of helpful 
information and content, 
including a detailed summary, 
video footage, and images of the 
moon landing that users can 
view and explore. This result 
would be very helpful for many or 
most users. 


This is an informational query on 
a fairly broad topic. Different 
users may be looking for different 
things, yet accuracy and 
credibility of the source are 
important. 


This LP is a news article 
published on a reputable news 
website. The MC of the article 
covers recent efforts to reform 
STEM education in the U.S., and 
it includes citations from expert 
sources. This result would be 
helpful for many or most users. 


This LP is on a well-known 
fact-checking and debunking 
website. The page calls this 
story "False", then provides a 
credible explanation of how the 
authors came to that conclusion, 
including history of similar urban 
legends. This result would be 
very helpful for users seeking to 
verify the credibility of this rumor. 


This LP is an opinion piece 
written by the Editorial Board of 
the largest newspaper in the U.S. 
state of Minnesota. The MC of 
the article expresses 
well-informed opinions about 
Fleck's upcoming tenure as head 
coach, and it would likely be of 
interest to many or most users 
when this query was issued. 
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Query: [batman lego 


movie review ‘The Lego Batman Movie' is the best Batman . . : , 
] movie since 'The Dark Knight' - LA Times - Los This LP is an in-depth review of 


papaia the queried movie by a film critic 


Los Angeles Times > movies > la-et-mn-... 


User Intent: Find reviews f f i t tabl Thi 
of the 2017 film The LEGO | © AMP-Feb 9,2017 - Justin Chang reviews The Lego ala reputable newspaper. IS 


Batman Movie, starring Will Arnett. result would be very helpful for 


Batman Movie on credible 
many or most users. 


news or entertainment 
sources. 
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13.4 Moderately Meets (MM) 


A rating of Moderately Meets is assigned to results that are helpful and satisfying for many users or very satisfying for 


some users. 


Moderately Meets results have fewer valuable attributes than Highly Meets results. Moderately Meets results should 
still “fit” the query, but they might be less comprehensive, less up-to-date, come from a less authoritative source, etc. 


Moderately Meets results are not low quality, out-of-date, or inaccurate. Moderately Meets results are generally average 


to good. 


13.4.1 Examples of Moderately Meets (MM) Result Blocks 


Query and User Intent 


Query: [shutterfly] 


User Intent: Go to the 
Shutterfly website or find 
information about the 
company. 


Query: [tom cruise] 


User Intent: Find 
biographical information or 
current news/celebrity 
gossip about Tom Cruise. 


Query: [seattle, 
washington] 


User Intent: Find 
information, news, maps, 
etc. related to Seattle, 
Washington. 


Query: [city of angels] 


User Intent: Find 
information about the 
movie, song, musical, 
school, or some other 
entity by this name. 


Result 


Shutterfly | CrunchBase 
www.crunchbase.com/.../shutterfly 

Shutterfly is a manufacturer and digital retailer, 
offering personalized stationery and other related 
products and services 


Tom Cruise Pictures, Biography, 
Filmography, News ... - Starpulse.com 
www.starpulse.com/Actors/Cruise, Tom/ 

Tom Cruise Pictures, Biography, Filmography, 
News, Box Office, Videos, Awards 


i | a 
À Shelgne Wott 3 
LS L eCity 
iU We 
„University 
Silverdale Bainbridge -PIST 
Chico} Island 


y smerton Goa) 
7s 
i 


ir 


Seattle, WA 


DA RS Map daté ©2014 Google 


City of Angels Independent 


Studies School 

1449 S San Pedro St, Los Angeles, CA 90015 
& i © 
Call Directions Website 


Rating 


FailsM 


FailsM 


b 
FailsM 


i 
FailsM 
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SM 


MM 


z 


l 
FullyM 


| 
FullyM 


| 
FullyM 


l 
FullyM 


Moderately Meets 
Explanation 


The LP for this web result is a 
high-quality page on 
crunchbase.com—a member site 
on the TechCrunch 
network—with a profile of 
Shutterfly. The MC has a lot of 
helpful and accurate information 
about the company. This result 
would be helpful for many users 
or very helpful for some. 


The LP for this web result 
contains information about Tom 
Cruise, which would be helpful 
for many users or very helpful for 
some users. The photos and 
main links can easily be 
accessed on a phone. 


Note that there are many, many 
pages about Tom Cruise and this 
result is not helpful enough for a 
rating of Highly Meets. 


This is a broad query for a city. 
Different users may be looking 
for different things. 


The result shows a map of 
Seattle, Washington and clicking 
on the map brings up a more 
detailed map. This result may be 
helpful for some users. 


This result reflects one of several 
possible user intents, as the 
query is also the name of a 
movie, song, musical, and more. 
However, since the user location 
is near the school, this result 
would be very helpful for some 
users—specifically, users who 
are looking for the school. 
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Query and User Intent 


Query: [nicaraguan 
money] 


User Intent: Find out what 
Nicaraguan currency is 
called, what it looks like, 
the exchange rate, etc. 


Query: [where is virginia 
in the us] 


User Intent: Find where 
the state of Virginia is 
located in the U.S. 


Query: [every breath you 
take lyrics] 


User Intent: Find the 
lyrics to the song “Every 
Breath You Take,” which 
was written by Sting. 


Query: [kristen wiig] 


User Intent: Find more 
information about the 
actress, comedian, writer, 
and producer. 


Nicaraguan cordoba 


A FailsM 
Jicaragua, Currency ai 


Virginia is bordered by Maryland 
and Washington, D.C. to the north 
and east; by the Atlantic Ocean to 
the east; by North Carolina and 
Tennessee to the south; by 
Kentucky to the west; and by West 
Virginia to the north and west. 
Virginia - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/wiki/Virginia 


STING LYRICS - Every Breath You Take 
- A-Z Lyrics 
www.azlyrics.com/lyrics/sting/everybreat... 
Lyrics to "Every Breath You Take" song by STING: 
Every breath you take Every move you make Every 
bond you break... 


Kristen Wiig! The Official Website of 

Kristen Wiig. 

kristenwiig.com 

Mobile-friendly - The official website of Kristen 

Wiig. Kristen Wiig is not on Twitter, Facebook FailsM 
MySpace, or any other social networking 
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Moderately Meets 
Explanation 


This result displays the name of 
Nicaraguan currency. Some 
users may have their need met 
by learning the name of the 
currency, but many users may 
want images, know the exchange 
rate, etc. This result would be 
helpful for many users or very 
helpful for some users. 


The description in this result 
block may be helpful for users 
who are already familiar with this 
area in the United States. Many 
or most users would probably 
want to see a map or a clearer 
description. 


The LP for this lyrics website 
page has the requested lyrics. 
There are many lyrics websites 
on the web and many pages are 
not 100% accurate. Moderately 
Meets is an appropriate rating for 
average pages with the 
requested lyrics. 


This is Kristen Wiig’s official 
website, but the only content is 
links to her talent and PR 
agencies, along with a note that 
says she is not on any social 
networking sites. While this 
information is helpful, many 
users would want to see more. 
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13.5 Slightly Meets (SM) 


A rating of Slightly Meets is assigned to results that are helpful and satisfying for some or few users. Slightly Meets 
results may serve a minor interpretation, be low quality, have stale or outdated information, have some minor 
inaccuracies, be too specific, too broad, too niche for the general user population considering the query, etc. to receive a 
higher rating. 


Important note: Many users decide which result to click or tap on based on the title of the web result on the search 
results page. (Note that these titles usually come from webpages.) A result with a very misleading or exaggerated title 
should be rated Slightly Meets or lower, due to the poor user experience that occurs when the landing page does not 
match the expectation of the user when clicking or tapping on the result. 


13.5.1 Examples of Slightly Meets (SM) Result Blocks 


Query and User Intent | Result Rating Slightly Meets Explanation 


Query: [britney spears] The LP of this web result has a 


2006 article about Britney Spears 


Britney Spears Files for Divorce | 
User Intent: Find TMZ.com i i i | 


7 à j www.tmz.com/.../britney-spears-files-for-. FaileM SM MM M Fully filing for divorce. This is very old, 
information about Britney 7 - Britney Spears has filed a petition for a n H ully stale news making the result 
Spears (e.g., current divorce from Kevin Federline. TMZ obtained the legal helpful f i 

news, pictures) papers, ... elpful for tew users. 


Although the LP has 


Query: [honda odyssey] comprehensive information from 


2010 Honda Odyssey - Kelley Blue 


Book a reputable source, the 
User Intent: We will www.kbb.com/honda/odyssey/2010-hon... i i i information is about the 2010 
assume that users are lèw byi KBB:comEd SEN aes am sia HM FuM Honda Odyssey. This 
interested in the current Lear more’about the 2010|Honda Qayssey with information would be considered 
Kelley Blue Book expert reviews. Discover A 
Honda Odyssey model, A stale for the query today, making 
ca > information including j 
unless specified otherwise. this result helpful for some or few 
users. 
The LP of this web result is about 
the movie “Hot Dog,” which came 
Query: [hot dog] , out in 1984 and is a minor 
Hot Dog... The Movie (1984) - IMDb interpretation of this query. 
User Intent: Find N EE R TE Ta i —_— | | Because this is an unlikely 
š i irected by Peter Markle. With David Naughton, J s ` 5 
information about hot Patrick Houser, Tracy Smith, John Patrick Reger — - = mm -FulyM) interpretation, this web result 
dogs, such as recipes or Naive corn-pone ... would be helpful for few users, 
nutrition information. even though the landing page 
has good content from a 
reputable source. 
_ a. This result provides contact 
Query: [motorcycles] 2 Ai information for one of the two 
} motorcycle shops in Potsdam, 
User Location: Potsdam, _/Osovies Cycle shop 5 New York. It is possible this 
New York See cs b p | would be helpful for some users, 
Sovie's Cycle Shop t d i 
to reviews ee = rans on _ = FulyM | but many users would be looking 
User Intent: Find for information. Additionally, 
information about Ske rane Doy Meal there is distracting 


motorcycles. See information (a listing for an 


insurance agent). 
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This is a Wikipedia page that has 
List of Presidents of the United birthdays for all U.S. presidents, 
Query: [abe lincoln’s States by date of birth - Wikipedia including Abraham Lincoln. 
birthday] However, his birthday is not 
. . Mobile-friendly - The following is a list of prominently displayed and itis 
User Intent: Find this U.S. Presidents, organized by date of not obvious that the answer is 
specific piece of birth, plus additional lists of birth related there—users have to do some 
information. statistics work and search around on the 
United States Presidents by page to find the answer, making 
SM to SM+ an appropriate rating. 


en.m.wikipedia.org/.../List_of_Presidents.. 


The quality of writing in this 

Query: [lack of sex and , , article, which was created by a 

problems with my 5 Tips to Fix a Sexless Marriage Or person without expertise in 
Relationship - Ezine Articles marriage or relationship 


marriage] ezinearticles.com > ... > Marriage p= . 
Mobile-friendly - Feb 25, 2008 - Look to the past - FailsM counseling, IS poor. Even though 
Sexless marriages don't just spring up out of thin the article is about the query, the 
air. They are just symptoms of other ... page is low quality and 
untrustworthy. It would be helpful 


for few users. 


User Intent: Find help for 
marital issues. 


Stevens Creek Blvd 


Santana Row (@) 


Judro Way 


: [nai Burke Williams 
Query: [nail spa] 4.2 kkk k 16 reviews Burke Williams is a day spa that 


Day Spa offers many treatments like 

— massage therapy, skin care, etc. 
While it does have nail care 
ECO ee options, it is not a dedicated nail 
spa. 


User Location: San Jose, 


California > © FailsM 


User Intent: Find a nail 
spa in or near San Jose. 355 Santana Row #2010, San Jose, CA 
9 95128 


© Open today 8:00 am — 10:00 pm v 


More about Burke Williams 


Although there are a 

handful of species that 

uery: [what type of bles P ‘ 

Q [ yP live in freshwater, we tend pp This result block does not 


sharks live in rivers] to think of sharks as being . . : 
ocean dwellers. But there is also a contain enough information to be 
very unique shark that can live in | fully satisfying and mentions only 


sea aaa le wnat both ocean and fresh water FaisM one type of shark. Most users 
estuaries and lakes: the Bull Shark. would need to do further 


types of sharks live in Shark Savers :: How Bull Sharks none 
rivers. Survive in Fresh Water : 
www.sharksavers.org/.../how-bull-sharks- 


The answer in this result block is 
Most of Bexar County hi ; 7 
stayed in 210, while the =. -. really confusing. Even if users 
San Antonio suburbs and ~-=- > can figure it out, they would likely 
Query: [210 area code] oe tol Coy ai still have to click on the link, or 
were split off as area code 830. The t th bsite. t fi 
F southern portion became area code go 0 ano er website, to contirm 
ser Intent: Find wha 956. The 1997 split made San e information. 
U Intent: Find what the inf ti 
area uses the 210 area signed i Santis 
are split between two area codes. . . 
code for phone numbers. k While there is a map, you would 
Area code 210 - Wikipedia, the free B A 
encyclopedia have to expand or click on it to 
EE AOPEN ODAVATE COCE. RID see the details, so it is not that 


helpful. 


FailsM 
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Query and User Intent 


Query: [ibm] 


User Intent: Go to the 
IBM website or find 
information about the 
company. 


Query: [where is nibiru] 


User Intent: Find the 
location of "Nibiru", which 
was a fictional planet in 
the movie Star Trek Into 
Darkness, as well as a 
pseudoscientific outer 
planet in the solar system 


that doesn't actually exist. 


I 
FailsM 


> More images for ibm 


Planet Nibiru has appeared in the 
sky and DOOMSDAY is on the way 


https://www.thesun.co.uk/.../planet-ni 
biru-has-appeared... $ 
Aug 23, 2016 - Now a YouTube channel FailsM 
called Nibiru/Planet X has uploaded 

footage shot in August which allegedly 

shows a blood red moon alongside a ... 
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I 
FullyM 


l 
FullyM 


Slightly Meets Explanation 


This block contains images of the 
logo for IBM, but these images 
are not particularly helpful for this 
query. 


The title of this article makes it 
appear as though the article 
describes an impending 
catastrophe. Yet, the MC states 
in its third paragraph that Nibiru 
"doesn’t actually exist" and calls 
believers in it "conspiracy 
theorists". The title contradicts 
the content of the article, and as 
a result could mislead users who 
don't read the full article. 


Note: Many users decide which 
result to click or tap on based on 
the title of the web result on the 
search results page. Results 
with very misleading or 
exaggerated titles should be 
rated Slightly Meets or lower, 
due to the poor user experience 
that occurs when the landing 
page does not match the 
expectation of the user when 
clicking or tapping on the result. 
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13.6 Fails to Meet (FailsM) 


A rating of Fails to Meet should be assigned to results that are helpful and satisfying for no or very few users. Fails to 
Meet results are unrelated to the query, factually incorrect (please check for factual accuracy of answers), and/or all or 
almost all users would want to see additional results. These results completely fail to meet the user intent, such as a lack 
of attention to an aspect of the query (or user location) that is important for satisfying user intent. Fails to Meet may also 
be used for results that have very stale or outdated information. 


Search results should never surprise people with unpleasant, upsetting, offensive, or disturbing content. For this reason, 
all of the following types of content should be rated Fails to Meet if it is clear that the user is not looking for such content: 


Harmful to Self or Other Individuals (Section 4.2) 
Harmful to Specified Groups (Section 4.3) 
Harmfully Misleading Information (Section 4.4) 
Untrustworthy (Section 4.5) 
Spammy (Section 4.6) 
Porn (Section 15.1) 


13.6.1 Examples of Fails to Meet (FailsM) Result Blocks 


Query and User Intent 
Query: [dogs] 


User Location: Pittsfield 
Charter Township, 
Michigan 


User Intent: Find 
information (e.g., pictures, 
breeds, training details, 
etc.) related to dogs. 


Note: For this query, it is 
very unlikely users want to 
go anywhere in person. 


Query: [mike] 


User Location: Eustis, 
Florida 


User Intent: Find 
information about 
something related to Mike. 


Note: For this query, it is 
unlikely users want to go 
anywhere in person. 


Result 


Gre ot 

N Ò Kor oos pa care * © 
= e fo 

< TO 


Amp Bow Wow 
o 


© sunset Ahimal Care LLC 


Arbor Dog Day Care 
45 kkk (7) 


Pet Sitte 


Camp Bow Wow 


4.5 eke we (38 


D Da 


Sunset Animal Care LLC 
No reviews 


Mike Bender Golf Academy 
No revi 


Private Golf Cot 


Closed now 


se - Shadowm 


Mike Holt Enterprises, Inc 
re 


2 review 
Publisher 


Closed now 


Parkwa’ 


Rating 


FailsM 
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| 
FullyM 


| 
FullyM 


Fails to Meet Explanation 


This is a broad informational 
query for an animal. Different 
users may be looking for different 
things. 


This result block shows 
visit-in-person information for 
three different dog care providers 
near the user location. However, 
the query is very broad and it is 
unlikely users want to go 
anywhere in person. 


This is a broad query and it is 
unclear exactly what the user is 
looking for. 


This result block shows 
visit-in-person information for two 
businesses that contain the 
name Mike. However, the query 
is very broad and it is very 
unlikely these businesses are 
what users seek given the query. 
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Query: [tampa rays] 


User Location: Tampa, 
Florida 


User Intent: Find 
information (e.g., Scores, 
roster, recent news) about 
a professional baseball 
team in Florida called 
Tampa Bay Rays. 


Note: For this query, it is 
unlikely users want to go 
anywhere in person. 


Query: [wral] 


User Location: Wake 
County, North Carolina 


User Intent: Go to the 
WRAL website or find 
other information about the 
news station in North 
Carolina. 


Note: Users generally 
would not go to a news 
station in person. 


Query: [texas farm 
fertilizer] 


User Location: Dallas, 
Texas 


User Intent: Find 
information about, or 
where to buy, fertilizer from 
the company Texas Farm. 


DIRECTIONS WEBSITE 


200 Madison St, Tampa FL 33602 


© closed now Hours v 


More about Bob’s Cards 


42 2 kkhk) 


Auburn Tower 


Texas Farm Products Co 
Farm 


DIRECTIONS WEBSITE FailsM SM 


915 S Fredonia St, Nacogdoches, TX 
75964 


iv] Claim this business 


More about Texas Farm Products Co 
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This result block shows 
information to visit a sports card 
store in person, which happens 
to have the baseball team in the 
name. However, remember that 
for this query, it is unlikely users 
want to go anywhere in person. 


Note: This business was 
fabricated for the purposes of 
this example, and does not exist 
in real life. 


WRAL is a television news 
station in North Carolina. Users 
want to go to the website or find 
information about the station. 


In this example, clicking on the 
blocks show information about 
the businesses’ corporate 
headquarters. Note that 
television stations (like radio 
stations, newspapers, etc.) 
usually have special contact 
information for the public that is 
different than their corporate 
headquarter information. Also, 
with the query just being the 
name of the station, it is very 
unlikely that users want this kind 
of information. 


This result block shows the 
Texas Farm corporate office, 
which is not helpful given the 
user intent—users could not 
physically go to the company’s 
corporate office to buy or learn 
about the product. Fertilizer is 
the kind of product that you 
would have to buy at a store. 
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he a ional = y 
Er RNN 


Query: [united 656] 


User Intent: Find 
information about flight 
656 operated by United 
Airlines (e.g., whether the 
flight is on time, what gate 
it is departing from). 


Query: [german cars] 


User Intent: Find 
information about German 
cars or go to the official 
homepage of a German 
automaker. 


Query: [company to get 
rid of the possum in my 
attic] 


User Location: 
Naperville, Illinois 


User Intent: Find a 
company to trap and 
remove a possum from the 
attic. 


Query: [starting jets 
quarterback 2001] 


User Intent: Find the 
name of the starting 
quarterback for the New 
York Jets football team in 
2001. 


Query: [doctor salary] 


User Location: San 
Francisco, California 


User Intent: Find 
information about doctor 
salaries. 


Second Unitarian Church 
1 review 


Church - W Barry Ave 


O'Hare International Airport 

3.6 kkkh (1,007) 

Airport - W O'H 

Massive airport serving the Midwest 


Catalyst Ranch 
46 kkkkk (11) 
Event Venue - W Randolph St 


Subaru of America: Subaru Cars, 
Sedans, SUVs, Crossovers, Wagons 
https://www.subaru.com/ 

Visit Subaru of America for reviews, pricing and 
photos of Subaru Cars, Sedans, Wagons, 
Crossovers & SUVs. 


Complete Pest Control - Adelaide 
South Australia | Adelaide Pest ... 
completepest.com.au/ 

Our family owned business has been delivering 
professional pest control services to metropolitan 
Adelaide since 1988 

Termite Treatments - Pest Specials 


E https:/www.espn.com > name > nyj 


New York Jets Roster | ESPN 


Visit ESPN to view the New York Jets team roster for 
the 2014 season. 


FailsM 


FailsM 


FailsM 


FailsM 


In 2013-2014, annual tuition and 
fees at public medical schools 
averaged approximately $31,783 for 
state residents and $55,294 for non- 
residents. At private schools, tuition 
and fees averaged $52,093 for 
residents and $50,476 for non- 
resident students. 


How Do I... Pay for Medical School? - 
Pay for Med School - Aspiring ... 
https://www.aamc.org/.../pay-med-school... 


Feedback 


1 
FailsM 
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This result shows information for 
a church, an event venue, and 
the closest airport. However, 
users clearly want to know 
details about a specific flight, and 
there is absolutely no information 
in the block about the flight. This 
result Fails to Meet the user 
intent. 


This LP is the homepage of 
Subaru, a Japanese car 
company, not a German car 
company. 


The page completely Fails to 
Meet the user intent and would 
be helpful for no or very few 
users. 


This LP is the homepage of a 
pest control company in 
Australia. U.S. users would need 
a U.S. company to take care of 
this problem. There is a 
mismatch between the page and 
the locale that makes this result 
helpful for no users—it 
completely Fails to Meet the 
user intent. 


Although this is a trustworthy 
website for information about 
NFL football, this LP does not 
contain the information 
requested by the user. This 
result completely Fails to Meet 
the user intent. 


The answer in this block is about 
the cost of education, not salary, 
which is misleading and doesn’t 

answer the user’s query. 
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Query: [stony child 
development center west 
point phone number] 


User Location: West 
Point, New York 


User Intent: Find the 
telephone number for the 
Stony Child Development 
Center in West Point, New 
York. 


Query: [who invented 
stairs] 


User Intent: Find out 
about the origin of stairs. 


Query: [batman] 


User Location: Anaheim, 
California 


User Intent: Find 
information about the 
fictional superhero that 
appears in American 
comic books, movies, and 
television shows. 


Query: [obama] 


User Location: Austin, 
Texas 


User Intent: Find 
information about Barack 
Obama. 


(845) 446-5504 


West Point Club, Phone 


“Stairs have become such a commonplace 


fixture in contemporary architecture that it is 


easy to forget that they were not invented 
until 1948, by Swiss architect Werner 


Bésend@rfer.” Sep 9, 2007 


history of the stairs | cghm 
www.cghm.org > wow > history-of-the-st... 


i 
j İkiztepe 
Batman 
Site 
Erkökli 
Akça 
Kuyubaşı Map data ©2015 Basarsoft, Google 


FailsM 


Balpinar 


Batman/Batman Province Turkey 


FailsM 
Weather: 50°F (10°C), Wind NW at 4 mph (6 
km/h), 97% Humidity 


Hotels: 3-star averaging $110. View hotels 


v More about Obama 
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The answer in this block provides 
the phone number for the West 
Point Club, not the Stony Child 
Development Center located in 
West Point, New York. This 
result Fails to Meet the user 
intent. 


The answer provided in this 
result block is factually 
inaccurate. Stairs have been a 
common architectural feature of 
buildings, pathways, outdoor 
structures, etc. since long before 
1948, and they were not invented 
by a person named Werner 
Bösendörfer. 


It is extremely unlikely 
(potentially a no-chance 
interpretation) that this query is 
looking for information on a city 
in Turkey called Batman, given 
that the user is located in the 
United States. No or almost no 
users would be satisfied with this 
result. 


It is extremely unlikely 
(potentially a no-chance 
interpretation) that this query is 
looking for information on a city 
in Japan called Obama, given 
that the user is located in the 
United States. No or almost no 
users would be satisfied with this 
result. 
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Query: [weather paris, 
texas] 


User Location: Paris, 
Texas 


User Intent: Find current 
weather information for 
Paris, Texas. 


Query: [what is the third 
quarter earnings for ge] 


User Intent: Find this 
most recent third quarter 
earnings information for 
General Electric. 


Query: [city of angels] 


User Location: 
Chicago, Illinois 


User Intent: Find 
information about the 
school, a movie by this 


name, or some other entity 


by this name. 


Query: [go kart for sale] 


User Location: 
Warrington, Pennsylvania 


User Intent: Find go karts 
for sale in the Warrington, 
Pennsylvania area. 


Paris, France 
Thu, 12 AM, Clear 


°F °C Precip 
®© Humidity: 
Wind: 3 m 


E- j e 
7AM 12PM SPM 10PM 


SAT SUN MON 


General Electric Com... 
NYSE: GE - Jan 9 11:54 AM ET 


27.02 -0.19 (-0.70%) 
—— 


26.75 
10am 


1d 
Open 


High A ol 
Mkt Cap 


g Š ‘4, 
me S fide, daté ©2014 Google, 


City of Angels Independent 
Studies School 


4 reviews 
1449 S San Pedro St, Los Angeles, CA 90015 
(213) 745-1100 


Ss Y © 


Directions Website 


Happy Tymes Family Fun Center 
3.3 Kk KK * (37) 


Batting Cage Center - County Line Rd 


Speed Raceway 

3.9 khkk & (24) 

G Track - Blair Mill Rd 
ntil 11:00 pm 


Speed Raceway 

4.0 kkk kk (16) 
Go-Kart Track - US-130 
Open until 11:00 pm 


More go kart racing 


FailsM 


FailsM 


FailsM 
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This result completely fails to 
satisfy the user intent, which is to 
find weather information for 
Paris, Texas, and not Paris, 
France. No users would be 
satisfied with this result. 


This result does not answer the 
user’s question to show third 
quarter earnings for GE. No 
users would be satisfied with this 
result. 


This result reflects one of several 
possible user intents, as the 
query is also the name of a 
movie, song, musical, and more. 


Since the user location is in 
Chicago, very few or no users 
would be interested in seeing 
information about a school by 
this name in Los Angeles that is 
not well known outside of this 
city. It is very unlikely that people 
outside of Los Angeles would 
use this query to search for this 
school. 


This result block shows a go kart 
arcade in Warrington, 
Pennsylvania, and two other go 
kart arcades in the surrounding 
area. These arcades are places 
to go ride go karts, not purchase 
them, so the result completely 
Fails to Meet the user intent. No 
users would be satisfied with this 
result. 
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Query: [ralphs] 


User Location: San 
Clemente, California 


User Intent: There are 
two possible user intents: 
most users probably want 
to visit a nearby Ralphs 
location or go to the 
website. 


Query: [amazon] 


User Location: Austin, 
Texas 


User Intent: Go to the 
Amazon website. 


Query: [what is the closest 
large city] 


User Location: Myrtle 
Beach, South Carolina 


User Intent: Find the 
closest large city to Myrtle 
Beach, South Carolina. 


Query: [australian open 
mens singles result 2008] 


User Intent: Find a page 
that displays the 2008 
men’s singles result for 
this tennis tournament. 


Query: [tooth loss five 
years old] 


User Intent: Find 
information about tooth 
loss in a five-year-old 
child. 


DIRECTIONS 


B Ralphs 
Friars Mission Cente © 


DIRECTIONS 


© 


DIRECTIONS 


Merc 


Amazon Campus = 


ner Moe st Republ 


Harrison St Harri: 


orate Office 


DIRECTIONS 


9 440 Terry Ave N, Seattle. WA 98109 


© Open today 7:00 am - 5:00 pm v 


More about Amazon.com 


News for large city 


County expanding use of large 
recycling bins, which city 
continues to study 

The State - 12 hours ago 


Murray, Sawant sworn in before large crowd... 
The Seattle Times (blog) - 8 hours ago 


2004 Australian Open — Men's Singles - 
Wikipedia, the free ... 
en.m.wikipedia.org/.../2004 Australian 
Mobile-friendly - Roger Federer defeated Marat Safin 
7-6(7-3), 6—4, 6-2 in the final to win the Men's 
Singles title at the 2004 Australian 

Seeds - Draw - References - External links 


Gallery of Pennsylvania Fishes - 
Chapter 16, Pikes and Mudminnows 
fishandboat.com/pafish/.../chap16pikes.ht... 
Pikes and Mudminnows. Family Esocidae. Family 
overview: Pike are long, slender, “duck-billed" 
predator fish, popular ... 
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FailsM 


FailsM 


FailsM 


FailsM 


FailsM 


FullyM 


FullyM 


FullyM 


FullyM 


FullyM 


Ralphs is a nationwide 
supermarket chain. These 
locations are all in San Diego, 
California, which is a major city 
south of the user location (about 
60 miles away). These results 
are too far to be helpful. 


Users issuing this query want to 
go to the Amazon website. 


Showing information about how 
to visit or call the corporate office 
for an online company would not 
be helpful. Very, very few people 
ever go to an Amazon corporate 
office, which is not open to the 
public. If someone needed to go 
to the office for an interview or 
business visit, they would need 
to get that information from 
someone at the company or 
would use a more specific query. 


A news result is inappropriate 
and completely unhelpful for this 
query. No users would be 
satisfied with this result. 


This LP is about the 2004 
Australian Open, not the 2008 
Australian Open. It does not pay 
attention to an aspect of the 
query that is important for 
satisfying user intent (i.e., 2008). 
The result Fails to Meet the user 
intent. 


This LP has information about 
tooth loss in pike fish and has the 
words “five years old” on the 
page. This result Fails to Meet 
the user intent because it has 
keyword matches only and would 
be helpful for no users. 
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Query: [what is wedding in 
spanish] 


User Intent: Find how to 
say the word “wedding” in 
Spanish. 


Query: [american express] 


User Intent: Go to the 
American Express card 
website or get information 
about the company and its 
products and services. 


Query: [how to quit 
smoking] 


User Intent: Find 
information on ways to quit 
smoking. 


Query: [starting order for 
today’s nascar race] 


User Intent: Find 
information about the 
starting order of the 
Nascar race that day. 


Query: [air canada phone 
number] 


User Intent: Find the 
customer service phone 
number for the airline Air 
Canada. 


Query: [obama news] 


User Intent: Find recent 
news about Barack 
Obama, the 44th president 
of the United States. 


manana 


Translate "wedding" to Spanish 


The Lipstick Chronicles: November 


2007 


thelipstickchronicles.typepad.com/.../11/ 
Nov 30, 2007 - ... of infected machines to mount a FailsM 
brute force attack against, say, American Express 


or Wells Fargo. 


How to type a cigarette - ws.gy 


xe4c.ws.gy/fo 


The eGo is one of the most popular models in the 


FailsM 


e-cigarette industry. It's affordable, provides great 
battery life, has an . 


NASCAR Schedule 
Sprint Cup Series-Nationwide Series-Camping World Truck Series 


1 


Date 
Feb 16 
Feb 21 
Feb 21 
Feb 24 


2 Mar3 


1 (888) 247-2200 


Air Canada, Customer service 


Race 


The Sprint Unlimited 


WinnerTime 


5:29 PM 


Budweiser Duel 1 11:19 


Budweiser Duel 2 


Daytona 500 


1:00 PM 
10:30 


Subway Fresh Fit 500 
+ Show more 


FailsM 


A 


AIR CANADA 


More about Air Canada 


Obama Announces Plans For A Third 
Term Presidential Run - News ... 
newsexaminer.net/.../obama-announ 
ces-plans-for-a-third-+term.... 


Washington, DC — President Barack 


Obama shocked the country this 
morning with news that he is running 


for a third term. 


FailsM 
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The translation in the result block 
is completely wrong and 
inaccurate. 


Mañana means “tomorrow,” not 
“wedding.” 


The LP is a humorous blog post 
about a wife helping her husband 
to buy a suit. The page mentions 
“American Express,” but is 
insufficiently related to the query 
to be helpful for users, so it Fails 
to Meet the user intent. 


This LP has gibberish text. Read 
this sentence: “How do you make 
a cigarette symbol on the 
keyboard? In.” The quality of the 
landing page is so low that the 
page Fails to Meet the user 
intent. 


This result shows a NASCAR 
schedule, but does not have the 
starting order. The content is not 
what the user is looking for and 
Fails to Meet the user intent. 


Remember that you should 
check for the factual accuracy of 
answers. For this result, go to 
the company’s page at 
www.aircanada.com/en/customer 
care/int/ and select USA to check 
the number shown here. 
According to the airline’s website, 
the correct phone number is 
1-888-247-2262. Because this 
answer is incorrect, it should be 
rated Fails to Meet. 


The MC of this page is an 
example of a demonstrably 
inaccurate news report, including 
a fictitious quotation by Barack 
Obama. Because the content is 
presented as a factual news 
article, it is inaccurate with an 
intent to deceive users and 
should be rated Fails to Meet. 
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Query: [does smoking 
cause cancer] 


User Intent: Find out 
whether smoking causes 
cancer. 


Query: [miley cyrus] 


User Intent: Find 
information about the 
American singer/actress 
Miley Cyrus, such as 
biographical info, 
discography/filmography, 
current news, etc. 


Smoking does NOT cause lung 
cancer, in fact it just might protect ... 
https://www.sott.net/...-Smoking-do 
es-NOT-cause-lung-cancer-in-fact-... 
Apr 10, 1999 - The following excerpt 
is taken from Lauren A. Colby's book 
In Defense of Smokers. ._. 


—| 
FailsM FullyM 


TRAGIC: MILEY CYRUS DEAD AT THE AGE OF 22 - 
Now8News 
Now8News > miley-cyrus-dead-age-22-2 


. . 
MILEY CYRUS DEAD AT THE AGE OF 22, Hollywood, CA - FailsM FullyM 


Miley Cyrus’ lifeless body was found by her entourage in a 
bathtub at her Hollywood home on Wednesday, authorities 
revealed. ... Meanwhile, paparazzi are swarming her 
Hollywood home awaiting Cyrus’ body to emerge. 
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Despite strong medical 
consensus linking tobacco to 
lung cancer—which can be found 
at authoritative sources by 
searching [lung cancer and 
smoking]—this LP claims there is 
no connection between smoking 
and cancer by selectively 
interpreting some experiments 
done on animals. 


Additionally, this is a YMYL query 
on a medical topic, so accurate 
and trustworthy information is 
highly important. This page 
directly contradicts 
well-established medical 
consensus and may potentially 
cause harm to users, so it should 
be rated Fails to Meet. 


Miley Cyrus was alive at the time 
this example was written. This 
LP is an article written by an 
unknown author that was 
designed to look like a news 
story and falsely states that she 
died. Furthermore, the page is 
found on a low quality website, 
and the article has not been 
updated, clarified, or removed 
even days after having been 
proven false. This page is 
factually inaccurate and 
deceptive to users, so it should 
be rated Fails to Meet. 


Remember: Just because a 
website has the look and feel of 
a traditional news site does NOT 
mean that you should consider 
the reporting to be factually 
accurate without further 
research. Watch out for sites 
that present themselves as 
news, but were actually created 
with the intent to mislead or 
misinform users. 
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Query: [holocaust history] 


User Intent: Find 
historical information about 
the Holocaust. 


Query: [who is hitler's 
daughter] 


User Intent: Find the 
name of or information 
about Adolf Hitler's 
daughter, if Hitler in fact 
had a daughter. 


Query: [islam] 


User Intent: Find 
information about the 
religion of Islam. 


Top 10 reasons why the holocaust 

didn't happen. - Stormfront 

https://www.stormfront.org » General 

> History & Revisionism 

Dec 19, 2008 - 10 posts - 8 authors FailsM 
History & Revisionism Reexamining 

history, particularly the court 

historians’ .... The Holocaust Lie 

more than anything else keeps .... 


The Vatic Project: Angela Merkel is 
Hitlers Daughter and a Rothschild 
vaticproject.blogspot.com/.../angela- 
merkel-is-hitlers-daughter-and.htm! 
Dec 22, 2013 - Then on November 
22nd, 2005, Hitler's biological 
daughter, Angela Merkel was elected 
Chancellor of Germany. The day, ... 


Proof that Islam is Evil, Violent, and 
Intolerant- Straight From the Koran 
janmorganmedia.com/.../proof-isiam 
-evil-violent-intolerant-straight-koran/ 
May 13, 2014 - Every time | write and 
article that sheds Islam in a negative 
light, my life gets threatened .. 
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This query is seeking historical 
information. The LP of this web 
result is on a well-known white 
supremacist hate site (read more 
here or here). The discussion on 
this page is inaccurate and 
misleading, since the Holocaust 
is an extremely well-documented 
historical event. 


Remember: Pages that directly 
contradict well-established 
historical facts should be rated 
Fails to Meet, unless the query 
clearly indicates the user is 
seeking an alternative viewpoint. 


This LP is a blog post presenting 
a factually unsupported 
conspiracy theory that Angela 
Merkel is the daughter of Adolph 
Hitler. Because the MC is 
inaccurate and misleading, it 
completely Fails to Meet the 
user intent, even though the topic 
of the page matches the query. 


Note: A more helpful result for 
this query might confirm that 
Adolf Hitler had no children, or 
provide information on the novel 
Hitler's Daughter by Jackie 
French. 


This is a broad query seeking 
information about Islam. The LP 
was created for the purpose of 
stating the author's opinion that 
Islam is not a religion. The 
author lacks expertise in the 
topic; the writing has frequent 
misspellings, typos, and 
grammar errors; and the page 
seems to exist to promote 
intolerance or hate. 
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14.0 The Relationship between Page Quality and Needs Met 


The Needs Met rating is based on both the query and the result. You must carefully think about the query and user intent 
when assigning a Needs Met rating. 


j—— =} a pe | 


FailsM SM MM HM FullyM 


The Page Quality rating slider does not depend on the query. Do not think about the query when assigning a Page 
Quality rating to the LP. 


O ee ee, 


WA Lowest Medium Highest 


Some results don’t have a Page Quality slider. If a result block has no Page Quality rating slider, you do not have to give 
a Page Quality rating. If there is a Page Quality slider, please assign a Page Quality rating based on the landing page. 


Here is some guidance about assigning Needs Met and Page Quality ratings: 


e Useless results should always be rated FailsM, even if the landing page has a high Page Quality rating. Useless 
is useless. 


e On-topic, helpful, but low Page Quality results should get lower Needs Met ratings than on-topic, helpful, and high 
Page Quality results. The Needs Met scale encompasses all aspects of “helpfulness,” and many users find low 
Page Quality results less helpful than high Page Quality results. Your ratings should reflect this. 


e The HM rating should be given to helpful, high Page Quality pages that are a good fit for the query. The HM 
rating may also be used for results that are very helpful, medium quality, and have other very desirable 
characteristics, such as very recent information. 


e The HM rating may not be appropriate if a page has low Page Quality or has any other undesirable 
characteristics, such as outdated or inaccurate information, or if it is a poor fit for the query. We have very high 
standards for the HM rating. 


e SM is often an appropriate rating for low quality but on-topic pages. However, a page can have such low Page 
Quality that it is useless for nearly all queries. Gibberish pages are a good example of pages with low Page 
Quality that should be rated FailsM. An exception to this is queries with clear website intent, where the target 
website should be rated FullyM even if the page has low Page Quality. 


e Remember that if a page lacks a beneficial purpose, it should always be rated Lowest Page Quality - regardless 


of the page’s Needs Met rating or how well-designed the page may be. Please review Section 4.1 for a summary 
of other types of Lowest Page Quality pages. 
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Query and User Intent 


Query: [how do electric 
cars work] 


User Intent: Find 
information about how 
electric cars work. 


Query: [symptoms of 
dehydration] 


User Intent: Find 
information about the 
symptoms of 
dehydration. 


Electric Car Motors - How Do Electric 
Car Motors ... - Ezine Articles 
ezinearticles.com/?...Motors...Motors... 
Mobile-friendly - Oct 10, 2008 - This engine which 
runs on gasoline principally has dominated the 
world of automobiles until recently. 


Mobility scooter - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/wiki/Mobility_scooter 
Mobile-friendly - ... configured like a motorscooter. 
It is often referred to as a power-operated 
vehicle/scooter or electric scooter as well. 


FailsM FullyM Lowest 


Symptoms of Dehydration 
www.symptomsofdehydration.com/ 

What are the symptoms of dehydration? Learn 
about the signs you can expect when dehydration 
begins and also as it... 


FailsM 


Dehydration Symptoms - Diseases and 
Conditions - Mayo Clinic 
www.mayoclinic.org/.../dehydration/.../sy... 
Mobile-friendly - Severe dehydration, a medical 
emergency, can cause: * Extreme thirst + Extreme 
fussiness or sleepiness in infants and children; 
irritability and confusion in adults * Very dry mouth, 
skin and mucous membranes * Little or no 
urination — any urine that is produced will be 
darker than normal * Sunken eyes 8 more items 


FailsM FullyM Lowest 
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Medium 


Medium 


Highest 


Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


Needs Met: This page would be 
helpful for few users. 


Page Quality: Read this content 
out loud. The article on the LP is 
shallow and unintentionally 
humorous. The MC has four 
paragraphs of text, but it basically 
says that electric cars use 
electricity and have to be 
charged. The page also lacks 
E-E-A-T. Low is an appropriate 
rating. 


Needs Met: The query asks how 
electric cars work, but the article 
on the LP is about mobility 
scooters. 


Page Quality: There is a lot of 
helpful MC on the landing page 
on Wikipedia.org. The page has 
high E-E-A-T. Medium+ to High 
is an appropriate rating. 


Needs Met: Even though the 
article is topical, the information is 
not reliable and potentially 
misleading or dangerous. 


Page Quality: This is a YMYL 
topic. This page has no evidence 
of medical expertise/authority and 
is not trustworthy. 


Needs Met: This result would be 
very helpful for most users. 


Page Quality: This is a YMYL 
topic. This page is on a highly 
authoritative medical website and 
has a lot of reliable and accurate 
MC. This page is very 
trustworthy. 
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Query and User Intent | Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


3 Oct: 4 Oct . j 
a toe) (Ce Needs Met: This block shows the 
number of octaves on a piano, not 
a guitar. Even though this SCRB 
seven octaves has a link to a high quality landing 
Query: [how many page about pianos, it is irrelevant 
octaves on a guitar] An 88-key piano has seven octaves plus three lower to the query. Useless results are 
notes (B, B flat and A) below the bottom C. It has 52 useless. This result Fails to Meet 
E; white keys and 36 black keys (sharps and flats), with + 
User Intent: Find out the each octave made up of seven white keys and five the user intent. 
number of octaves ona black keys. Mar 28, 2020 
guitar. is Chassis fds viainimertas plans O Page Quality: There is a lot of 
Why do pianos have 88 keys? - Classic helpful MC on the landing page. 
FM The page has medium to high 
E-E-A-T. Medium+ to High is an 
appropriate rating. 


mM Feedback 


p | 


Medium Highest 
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15.0 Rating Porn, Foreign Language, and Did Not Load Results 


You will be asked to assign Porn, Foreign Language, and Did Not Load flags to result blocks when appropriate. Some 
rating tasks may also ask you to identify Upsetting-Offensive and/or Not-for-Everyone results. 


All flags should be assigned based on the result alone and do not depend on the query. Here is a screenshot of the flags: 
Porn: No Foreign Language: No; Did Not Load: No 


Click on the flag name to select it. The flag will turn red and change the “No” to “Yes.” For example, here is a result block 
that shows when the Foreign Language flag should be used. 


Query and User Intent | Result Block and LP, Needs Met Rating, Flag Explanation 


A E This LP is the homepage 
www.baidu.com/ of the Baidu website. 


Pero ; The result is the target 
PIRNER- ESRDE. SA BR. Hig pagea the query. butis 


Query: [baidu.com] 


Locale: English (US) Fo BR AWE FWAR RIRE. in a foreign language 
(Chinese). 

User Intent: Go to the i n a 

Baidu website. FailsM SM MM HM FullyM This result merits a 
FullyM rating and the 


Porn: No | Foreign Language: Yes | Did Not Load: No Foreign Language flag. 


15.1 Porn Flag 


Please assign the Porn flag to all porn pages, whether the query is porn-seeking or not. The Porn flag does not depend 
on the query or user intent. 


This flag should be used if the content inside the result block, or the content on the LP, is pornographic, including porn 
images, links, text, pop-ups, and/or prominent porn ads. An image may be considered porn in one culture or country, but 
not another. Please use your judgment and knowledge of your locale. 


Remember: The Porn flag is used to indicate that the result is porn. It doesn’t matter whether the query “asks for porn”. 


15.2 Needs Met Rating for Porn Results 


15.2.1 Needs Met Rating for Clear Non-Porn Intent Queries 


If the user intent is clearly not porn-seeking, a landing page that has porn for its MC should be rated Fails to Meet. 


e When the user intent is clearly not porn, a porn result should be considered unhelpful or useless. 
e Uninvited porn is a very bad experience for many users. 


The following queries should be considered non-porn intent queries: [girls], [wives], [mature women], [gay people], [people 


kissing], [boy speedos], [moms and sons], [pictures of girls], [pictures of women], [mothers and daughters], [cheerleaders], 
etc. 
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Remember to assign the Porn flag for all porn pages. Use your judgment and represent users in your locale. 


: : Example 
[toys], English (US) Find toys to buy Warning — this page is porn. 


Porn Flag? 


, Fam | Yes 

f ; Find the answer to this Example 

[how tall is a camel], English (US) question about camels. Warning — this page is porn. imo | Yes | 
: ; . . Example 

[car pictures], English (US) Find pictures of cars. Warning — this page is porn. Fats | Yes | 


15.2.2 Needs Met Rating for Possible Porn Intent Queries 


Some queries have both non-porn and porn interpretations. For example, the following English (US) queries have both a 
non-porn and an erotic or porn interpretation: [breast], [sex]. We will call these queries “possible porn intent” queries. 


For "possible porn intent" queries, please rate as if the non-porn interpretation were dominant, even though some or many 


users may be looking for porn. For example, please rate the English (US) query [breast] assuming a dominant health or 
anatomy information intent. 


: Find anatomy or health 
F Find anatomy or health Example 
[breasts], English (US) information about breasts. Warning — this page is porn. imo | Yes | 


15.2.3 Needs Met Rating for Clear Porn Intent Queries 


For very clear porn intent queries, assign a rating to the porn landing page based on how helpful it is for the user. Even 
though there is porn intent, the page should still be assigned a Porn flag. 


Do not simply rate all porn pages for porn queries as MM or HM. Even though the query is porn and the result is porn, the 
page must fit the query and be helpful to get a high Needs Met rating. 


Pages that provide a poor user experience, such as pages that try to download malicious software, should also receive 
low ratings, even if they have some images appropriate for the query. 


Example: 
http://www.freeones.com 
Warning — this page is porn. 


Navigate to the Freeones 


[freeones], English (US) homepage 


Example: 
http:/Awww.baberoad.com 
Warning — this page is porn. 


: - : : - F Example 
[anime sex pictures], English (US) | Find anime sex pictures. Warning — this page is porn. MM to HM 
F Find porn pictures of Example 
[cheerleader porn], English (US) cheerleaders. Warning — this page is porn. MIMO FHM Yes 


Navigate to the Freeones 
homepage. 


[freeones], English (US) 
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15.3 Reporting Illegal Images 
Child Pornography 


Note: When working on rating projects in any task location or locale, you must follow United States federal law, which 
considers child pornography to be illegal. 


Definition of Child Pornography 


An image is child pornography if it is a visual depiction of someone who appears to be a minor (i.e., under 18 years old) 
engaged in sexually explicit conduct (e.g., vaginal or anal intercourse, oral sex, bestiality or masturbation as well as 
lascivious depictions of the genitals), or sadistic or masochistic abuse. The image of sexually explicit conduct can involve 
a real child; a computer-generated, morphed, composite or otherwise altered image that appears to be a child (think of 
images that have been altered using “Photoshop”); or an adult who appears to be a child; and the image can be 
non-photographic (e.g., drawings, cartoons, anime, paintings, or sculptures) so long as the subject is engaging in sexually 
explicit conduct that is obscene. If it is indistinguishable from child pornography, it is child pornography. 


Even if the pornographic image depicts children in a literary (think of the famous book "Lolita"), political (think of political 
cartoons), artistic, or scientific context, please send the link to your employer/company as instructed below. 


Depiction of the genitals does not require the genitals to be uncovered. Thus, for example, a video of underage teenage 
girls dancing erotically, with multiple close-up shots of their covered genitals, or images of children with opaque underwear 
that focus on the genitalia could be considered child pornography. 

An image of a naked child (e.g., in the bathtub or at a nudist colony) is not considered child pornography as long as the 
child is not engaging in sexually explicit conduct, or the focus is not on the child’s genitalia. Visual depictions of adults who 
look like children (e.g., a 35-year-old man play-acting in diapers, or an obvious woman dressed as a school girl) are not 
child pornography. If you don't think it's a minor, it probably isn’t child pornography. However, if you cannot tell that the 
person in the image is over 18 (e.g., an under-developed 18-year-old whose body hair has been waxed), that is child 
pornography. 


Reporting Instructions 


Please report images containing child pornography (URLs only) as instructed by your employer/company. 


15.4 Foreign Language Flag 

15.4.1 Using the Foreign Language Flag 

Please do not assign the Foreign Language flag when the language on the landing page is one of the following: 
e The task language 
e A language that is commonly used by a significant percentage of the population in the task location 


e English 


For example, most Catalan-speaking users in Spain also speak Spanish. Therefore, for rating tasks in Catalan (ES), the 
Foreign Language flag should NOT be assigned to landing pages in Catalan, Spanish, or English. 
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Here is a screenshot of a result block that shows when the Foreign Language flag would be used. 


Query and User Intent | Result Block and LP, Needs Met Rating, Flag Explanation 


Be This LP is the homepage 
A of the Baidu website. 


www.baidu.com/ 
The result is the target 


Query: [baidu] 


Locale: English (US) Ro PARE FOWAPHSSRTN BE @ in a foreign language 
(Chinese). 

User Intent: Go to the , i i i I 

Baidu website. FailsM SM MM HM FullyM This result merits a 
FullyM rating and the 


Porn: No | | Foreign Language: Yes | Did Not Load: No Foreign Language flag. 


Important: 


e Please assign the Foreign Language flag even if you personally understand the language, but most users in 
your locale do not. 

e Please assign the Foreign Language flag based on the language of the landing page, not the appearance of the 
result block. 

e Please remember to flag all foreign pages with the Foreign Language flag, even if most users in your locale 
would expect or want a foreign language page for the query. 

e Sometimes it is difficult to determine what language the landing page is in. The LP may have multiple languages 
or no words at all. In these cases, try to represent users in your locale. Does it feel like a foreign language page? 
When in doubt, don’t use the Foreign Language flag. 


15.4.2 Needs Met Rating for Foreign Language Results 


You must assign a Needs Met rating for all result blocks in your task, even if the result blocks have a foreign language 
landing page. 


In most cases, pages you flag as Foreign Language should be rated FailsM, because they cannot be understood by 
most users in your locale and are therefore useless. Remember that if users in your locale can read the language, then 
you shouldn’t be using the Foreign Language flag. Occasionally, you will encounter helpful Foreign Language pages. 


If the query clearly indicates that most users would expect or want a foreign language result, then the Needs Met rating of 
the foreign language page should not be FailsM. For example, please assign the FullyM rating and Foreign Language 
flag for baidu.com if the query is [baidu.com], English (US). 


Videos are often an example where foreign language pages are helpful and desired. Think about user intent and what 
pages are good for users. If the query “asks” for a foreign language song, band, film, sporting event, etc., then a video of 
the song, band, film, sporting event, etc. is helpful since it can probably be understood or enjoyed even though it is in a 
foreign language. For these types of queries, foreign language results are often expected. 


If the video is someone talking about the song, band, film, or event, the result probably cannot be understood or enjoyed 
and should be assigned a FailsM rating and the Foreign Language flag. 


Note: If you are unable to evaluate the Page Quality rating of a Foreign Language result, you do not need to assign a 
Page Quality rating and can leave the slider at N/A. 
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Query and User Intent 


Query: [video of celine dion 
singing s'il suffisait d’aimer] 


Locale: English (US) 


User Intent: Watch a video of 
Celine Dion singing this song. 


Query: [alex c] 

Locale: English (US) 

User Intent: Watch a video or 
listen to a song by this German 


composer or find information 
about him. 


Query: [kasal, kasali, kasalo] 
Locale: English (US) 


User Intent: Watch a trailer of 


this Filipino film or find information 


about it. 


Result Block and LP, Needs Met Rating Explanation 


Celine Dion - S'il Suffisait D'aimer Live - 
YouTube 


© YouTube app 
Sep 2, 2011 
Open on youtube.com 


@ YouTube app 
Mar 26, 2011 
Open on youtube.com 


Kasal Kasalo Kasali - YouTube 
m.youtube.com/watc... 
E Apr 8, 2007 
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This video is just what English (US) users are 
looking for, even though the video is not in 
English. The language of the LP is mostly 
English (even though the video is in French), so 
the Foreign Language flag is not needed. 


The query is for the German composer, Alex C. 
The landing page has a video with one of his 
songs sung by Y-ass (misspelled as “Yass” in 
the YouTube title) in German. It would be 
helpful for some or many English (US) users 
who type the query, even though it is not in 
English. The language of the landing page isn’t 
clear. Use your judgment when deciding 
whether to assign the Foreign Language flag. 


The query is for “Kasal, Kasali, Kasalo,” a 
Filipino film. The LP is a clip from the movie in 
Filipino (Tagalog), the language spoken in the 
Philippines. It would be helpful for many 
English (US) users who type the query, even 
though it is not in English. Use your judgment 
when deciding whether to assign the Foreign 
Language flag. 


139 


15.5 Did Not Load Flag 
15.5.1 Using the Did Not Load Flag 
Did Not Load is used to indicate technical problems with the webpage that prevent users from viewing any LP content. 


Use the Did Not Load flag when: 


e The MC of the landing page is a web server or web application error message and there is no other content on 
the page: no navigation links, no home link, no SC, and no Ads. See here for a Wikipedia page on different types 
of error messages. 

e The landing page is completely blank: no MC, no SC, and no Ads. 


Assign the Did Not Load flag based on the landing page, not the result block. 


Here is an example of a Did Not Load landing page. You cannot tell that the landing page doesn’t load by looking at the 
result block. 


Query and User Intent Result Block and LP, Needs Met Rating, Flag Explanation 


Douglas Instruments Success 

in Protein ... 

www.douglas.co.uk/ This page displays a generic 404 
Query: [douglas instruments] Douglas Instruments designs and manufactures message. There is no MC, SC, or 


robots for protein crystallization. Our systems are 
setting new standards ... 


Ads on the page. 

User Intent: Navigate to the 

homepage of this website. — E | This result merits a FailsM rating 
| | | and the Did Not Load flag. 


i . 
FailsM SM MM HM FullyM 


Porn: No Foreign Language: No | | Did Not Load: Yes 


Here are screenshots of other types of landing pages that should be assigned the Did Not Load flag: DNL1, DNL2. 


You should not use the Did Not Load flag for: 


e Malware warnings, such as “Warning — visiting this site may harm your computer!” (example). 

e Pages that have removed or expired MC (e.g., expired classified listing, removed social media post, products or 
services unavailable). 

e Pages that are inaccessible because you need a subscription to view the MC. 


15.5.2 Needs Met Rating and the Did Not Load Flag 


All result blocks must be given a Needs Met rating. If the landing page truly doesn’t load, assign the Did Not Load flag 
and rate the page FailsM. True Did Not Load pages are useless. 


Sometimes the page partially loads or has an error message. Give Needs Met ratings based on how helpful the result is 
for the query. Error messages can be customized by the website owner and are part of a well-functioning website. 
Sometimes these pages are helpful for the query. 


Note: If you are unable to evaluate the Page Quality rating of a Did Not Load result, you do not need to assign a Page 
Quality rating and can leave the slider at N/A. 
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Query and User Intent Result Block and LP, Needs Met Rating, Page Quality Explanation 
Rating, Flags 


Boys’ Snow Boots & Winter Socks | 
Nordstrom 

m.shop.nordstrom.com > Kids > Boys 
Mobile-friendly - Shop online for boys' snow boots 
& winter socks at Nordstrom.com. Browse cold- 
weather boots, waterproof boots ... 


Query: [boys pink snow 
shoes] 


User Intent: Find 
information about or 
purchase boys’ snow 


1 | 
shoes. i 


FailsM MM HM FulyM Lowest Medium 


— | 


Highest 


Por: No. Foreign Language: No Did Not Load: No 


Bible Gateway 
https://www.biblegateway.com/ 
Mobile-friendly - Unlock your own digital study 
library on Bible Gateway. Browse Products. Our 


Query: [bible passages] Blog. Daily Bible-focused news, articles, ... 


User Intent: Find 
specific passages in the l , 4 
Bible. MM HM 


=| | | 


FullyM Lowest Medium Highest 


Porn: No, Foreign Language: No | Did Not Load: No 


http://malicious-site.url/ > broadway 
Query: [broadway © hip f 


tickets] 


Click HERE for free broadway tickets! 


... maclicious site - broadway - tickets phishing - special 
offer - malware warning ... 


User Intent: This is a 
Know query or Do 
query. The user wants 
to search prices and/or 
purchase tickets to a 
Broadway show in New i R — m | | | 
York City. Fully 


FailsM MM HM Lowest Medium Highest 


Porn: No Foreign Language: No | Did Not Load: No 


© _https://paranormalhangout.com > 


Query: [paranormal 
activity] 


Paranormal Activity | Discussion Board 


Welcome to our Paranormal Activity discusion board! 
Come discuss the film, the series, or your own personal 
encounters with paranormal ... 


User Intent: Learn more 
about the 2007 film 
Paranormal Activity or 
the associated film 


series. 
1 ' 1 I 
FailsM MM FulyM N/A Lowest 


Porn: No Foreign Language: No | | Did Not Load: Yes 


Medium Highest 
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Needs Met: The MC has an error 
message, but the LP has a lot of SC. 
However, the page has no information 
about boys pink snow shoes and is 
unhelpful for the query. 


Page Quality: This LP is on a well-known, 
reputable merchant website. Despite the 
error message, this page does a nice job 
of explaining the issue and providing a 
search box, including helpful search tips 
and customer service information. 


Do not assign the Did Not Load flag. 


Needs Met: In spite of the customized “No 
results found” message on the LP, it has 
links to all passages in the Bible, organized 
by book. It would be very helpful for most 
users. 


Page Quality: This page is a well-known, 
reputable reference website. In addition to 
links to all passages in the Bible, this error 
page provides a search box as well as a 
suggestion explaining how to refine 
searches. 


Do not assign the Did Not Load flag. 


Clicking on this result block leads to a 
malware warning. You can assume that 
the page is malicious without continuing to 
the page. As discussed in Sections 4.5.5 
and 13.6, you should assign a FailsM 
Needs Met rating and Lowest Page 
Quality rating. 


Do not assign the Did Not Load flag. 


This page cannot be reached because the 
site's server IP address was not found. 
There is no MC, SC, or Ads on the page. 


This result merits a FailsM Needs Met 
rating, and no Page Quality rating is 
required. 


Assign the Did Not Load flag. 
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15.6 Additional Flags in Some Rating Tasks 


Some rating tasks may ask you to identify Upsetting-Offensive and/or Not-for-Everyone results. 


Mark content that may be upsetting or offensive from the perspective of a typical user in your locale as 
Upsetting-Offensive, keeping in mind that people of all ages, genders, races, religions, and political affiliations use the 
Internet to understand the world and other points of views. 


Mark content that may be unpleasant or uncomfortable for some people in your locale (e.g., content that may not be 
appropriate in a public space, professional environment, or school) as Not-for-Everyone. 


16.0 Rating Queries with Multiple Interpretations and Intents 


Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user 
intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are 
basically referring to the same thing: the phone made by Apple, Inc. 


Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to 
find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations. 


When giving Needs Met ratings for results involving different query interpretations, think about how likely the query 
interpretation is and how helpful the result is. 


e A very helpful result fora dominant interpretation should be rated Highly Meets, because it is very helpful for 
many or most users. Some queries with a dominant interpretation have a FullyM result. 

e Avery helpful result fora common interpretation may be Highly Meets or Moderately Meets, depending on 
how likely the interpretation is. 

e Avery helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be 
interested in that interpretation. 

e There are some interpretations that are so unlikely that results should be rated FailsM. We call these “no chance’ 
interpretations. 


16.1 Rating Queries with Both Website and Visit-in-Person Intent 
Some queries have two possible strong intents: 
1. Go to the website intent: in order to, for example, find out information, buy something online, make a reservation, 
schedule an appointment, interact with customer support, or fulfill some other need that can be satisfied online 


2. Visit-in-person intent: user wants to visit the store, business, etc. in person 


We know the user intent is to accomplish one or the other, but it is unclear which one the user wants. For these queries, 
result blocks that only satisfy one intent should NOT get a Fully Meets rating. 
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Query and User Intent 


Query: [target] 


User Location: 
Jacksonville, Florida 


User Intent: There are 
two possible strong user 
intents: most users 
probably want to visit a 
nearby Target location or 
go to the website to shop 
online, research 
products, find prices, etc. 


Query: [dmv] 


User Location: 
Belmont, California 


User Intent: There are 
two possible strong user 
intents: most users 
probably want to visit a 
nearby DMV location or 
go to the DMV website 
to renew a license, pay a 
fee, find some 
information, etc. 


Result Block and LP 


T 


Open until 10:00 pm 


More target 


Target 

m.target.com 

Mobile-friendly - ... with $25 purchase or free 
same day store pickup on all online orders 


F 


Bair island 


XD Neptune B 
(a)-e Jackson 
E Ee 

Soe FBeact 


£ = | 


DIRECTIONS 


© 


DIRECTIONS 


He Me. ia 
TN E 7 


A DMV Office in San 
Mateo 


Mate 
Closed now 


DMV 
0 Brewster Ave 


Closed now 


DMV - State of California 
https://www.dmv.ca.gov 


California DMV Home Page is available for 
customers to check out publications, download 
forms, brochures, FAQs 
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Needs Met Rating and Explanation 


l 
FailsM FullyM 


The result block shows three popular 
Target locations in Jacksonville, with 
information that is especially helpful 
for users who want to visit the store. 
This fulfills the user intent to find a 
nearby Target location. 


bs ' | 
FailsM SM FullyM 


The result is very satisfying and fulfills 
the user intent to shop online or 
otherwise use the website. 


FailsM Fully 


The result block shows two nearby 
DMV locations, with information that is 
especially helpful for users who want 
to visit the locations. This fulfills the 
user intent to find a nearby DMV 
location. 


FullyM 


The result is very satisfying and fulfills 
the user intent to find information or 
otherwise use the website (e.g., to 
review a license, pay a fee). 
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Query and User Intent | Result Block and LP Needs Met Rating and Explanation 
Ye) 


1 1 
FailsM SM MM HM FullyM 


A Citibank 
250 University Ave 


Query: [citibank] Closed now 

7 B Citibank 
User Location: Palo 2401 EI Camino Real 
Alto, California Closed now 


The result block shows three nearby 
Citibank locations in the user location 
of Palo Alto. The information is 
especially helpful for users who want 
to visit the bank. This fulfills the user 
intent to find a nearby Citibank 
location. 


C Citibank 
User Intent: There are 608 San Antonio R 
two possible strong user Closed now 
intents: most users 
probably want to visit a More citibank 
nearby Citibank location 
or go to the website to Citibank: Sign On 
bank online. https://online.citibank.com U pa l s ' = I mae 


Mobile-friendly - Discover financial services 
tailored to your life from Citibank®. Learn more 
about our full range of banking products 

Citi® Credit Cards - Login - Sign On 


The result is very satisfying and fulfills 
the user intent to do online banking or 
otherwise use the website. 


17.0 Specificity of Queries and Landing Pages 


Some queries are very general and some queries are specific. Here are some examples that compare levels of specificity 
of English (US) queries: 


More Specific Query Even More Specific Query 
[dining room chair] [ikea “henriksdal” highback upholstered chair] 
[library] [harvard library] [harvard anthropology library] 


[interview questions] [interview questions for teachers] | [practice interview questions used for teach for america] 
[restaurants] [chinese restaurants] [takeout chinese restaurants in downtown Austin] 
[coffee shops] [starbucks] [red rock coffee mountain view] 


Results for specific queries are easier to rate on the Needs Met scale because we know more about what the user is 
looking for. Giving a Needs Met rating for results for general queries can be difficult. As always, your rating is based on 
how helpful the result is for the query, not the specificity fit. 


When the query is a broad category, such as [cafes] [restaurants] [hotels] [books] [tourist attractions in paris] etc., popular 


and prominent examples may be considered very helpful. Please do web research to help you understand what is 
popular and prominent in different locations. 
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Query and User Intent 


Query: [credit cards] 


User Intent: Users are 
probably looking to sign 
up for a credit card 
online, or want to 
research credit cards 
before signing up. 


Result Block & LP, Needs Met & Page Quality Ratings Explanation 


Credit cards & Rewards credit cards | 
Visa USA 
usa.visa.com/...cards/credit-cards/index.js... 
Mobile-friendly - Whether you are looking for a 
premium rewards card or a traditional credit card, 
Visa has something to fit your needs. 

Apply for a Credit Card - Apply 


| | 


Medium 


= 


Fully Lowest Highest 


Credit Cards and Credit Card Offers - 
Apply Online | Discover Card 
https://www.discover.com/credit-cards/ 
Mobile-friendly - See all the ways you can earn and 
enjoy credit card rewards with the Discover it credit 
card, the game changing cash ... 


FullyM Lowest 


Best Credit Cards of 2015: Reviews, 
Rewards and Offers - NerdWallet 
www.nerdwallet.com/the-best-credit-cards 
Mobile-friendly - NerdWallet's credit card experts 
rank the best credit cards out there. Offers include 
no fee cash back cards with up to 5%... 


| | 


FailsM Lowest Medium 


Union Plus Credit Card for Union 
Members 
www.unionplus.org/...credit/credit-card 
Mobile-friendly - The Union Plus Credit Card is 
designed for union members. The credit card offers 
financial hardship assistance, strike ... 


FailsM FullyM Lowest 
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Even though there are many options, a 
page for an individual company that offers 
different cards could be very helpful. 


Needs Met: This LP is more specific than 
the query, but could still be helpful for 
many or most users because Visa is a 
popular credit card company. 


Page Quality: Visa is a credit card 
company with high E-E-A-T that offers 
services for credit cards, banking, etc. and 
has a good reputation. Hight to Highest 
is an appropriate rating. 


Even though there are many options, a 
page for an individual company that offers 
different cards could be very helpful. 


Needs Met: This LP is more specific than 
the query, but could still be helpful for 
many or most users because Discover is a 
popular credit card company. 


Page Quality: Discover is a company with 
high E-E-A-T that offers services for credit 
cards, banking, etc. with a good reputation. 
High+ to Highest is an appropriate rating. 


This page offers a list of top credit cards in 
a variety of categories. 


Needs Met: This LP fits the query. It would 
be helpful for most users. 


Page Quality: This website helps 
consumers make money decisions. It has 
a good reputation, high E-E-A-T, and has 
been recommended by several major 
newspapers. High is an appropriate 
rating. 


This page describes a credit card that 
requires union membership. 


Needs Met: Since the credit card requires 
union membership, the page would be 
helpful for some users. 


Page Quality: This company is an expert 
on its own credit card, which is issued by 
HSBC Bank, a financial services 
organization with a good reputation and 
high E-E-A-T. High is an appropriate 
rating. 
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Query and User Intent 


Query: [hotels] 


User Intent: Users are 
probably planning a trip, 
but this query is very 
general and vague. 


Result Block & LP, Needs Met & Page Quality Ratings Explanation 


Hotels, Hotel Rooms, Discount Hotel 
Rates & Cheap Motels on Orbitz 
www.orbitz.com/hotels/ 

Mobile-friendly - Find discount hotels, motels, and 
cheap hotel rooms with hotel reviews on Orbitz. Get 
our lowest hotel rates on our... 


Lowest 


Marriott 

www. marriott.com/default.mi 

Mobile-friendly - Browse and book hotel rooms 
that suit your needs and style and start your next 
adventure with Marriott. 


FailsM FullyM Lowest Medium 


Emeryville Hotels near Berkeley CA | 
Courtyard Hotel in ... - Marriott 
www.marriott.com/hotels/.../oakmv-court... 
Mobile-friendly - 4 days ago - This Emeryville, CA 
hotel near Berkeley offers a modern lobby with 
flexible work spaces, free Wi-Fi ... 

Hotel Details - 5555 Shellmound Street ... 


FailsM FullyM Lowest Medium 
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= 


Highest 


= 


Highest 


This is a popular travel aggregator website, 
and the hotel page on the site can help 
users find hotels in the U.S. Users can 
read reviews, compare hotels, or make a 
reservation. 


Needs Met: This LP fits the query. It would 
be very helpful for most users. 


Page Quality: Orbitz is a popular website 
with high E-E-A-T and a good reputation. 
High is an appropriate rating. 


This is a very popular hotel chain with 
hotels available in the majority of the U.S. 
at many different price points. 


Even though the list of possible hotel 
chains is long, the homepage of an 
individual chain that offers different prices, 
features, and location options could be 
very helpful. 


Needs Met: This LP is more specific than 
the query, but it would still be helpful for 
many or most users because the Marriott 
is a popular chain of hotels. 


Page Quality: The Marriott website gives 
information on Marriott hotels, a popular 
chain of hotels. Marriott has a good 
reputation and is an expert on Marriott 
hotels, making the information on this page 
highly authoritative. High+ to Highest is 
an appropriate rating. 


This is the webpage of the Marriott 
Courtyard hotel in Emeryville, California. 


Needs Met: The LP is too specific for the 
query, but this is a well-known brand and 
users can navigate to other Marriott hotels 
from this page. Few users would find this 
page helpful. 


Page Quality: The Marriott website gives 
information on Marriott hotels, a popular 
chain of hotels. Marriott has a good 
reputation and is an expert on Marriott 
hotels, making the information on this page 
highly authoritative. High+ to Highest is 
an appropriate rating. 
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Query and User Intent 


Query: [target] 


User Location: 
Jacksonville, Florida 


User Intent: Go to 
target.com or find a 
nearby Target store. 


aa 
Mianta 

CO Neptune 
Neptune B 

eptune B 


DIRECTIONS 


© 


DIRECTIONS 
until 11:00 pm 


Target 
6331 elt Blvc > 


an DIRECTIONS 
until 10:00 prr 


More target 


, No Page Quality rating 


FailsM Fully required 


Target 
m.target.com 
Mobile-friendly - ... with $25 purchase or free 
same day store pickup on all online orders . 


Q 


Medium 


k= ' 
FailsM SM 


Find Store - Target 
m.target.com/store-locator/find-stores 
Mobile-friendly - find a store. Target; find a store ... 
search for a store ... -or- view all stores by state. 
Only show stores with: Apple Store 


Fully Lowest Medium 


Electronics : Target 
m.target.com/c/electronics/-/N-5xtg6 
Mobile-friendly - Buy electronics online from the 
top brands at Target, from TVs and cameras to 
computers and tablets. Free shipping on... 

TVs & home theater - Computers & Office 


FailsM HM Medium 


Copyright 2023 


p| 


jo 


Highest 


| 


Highest 


Result Block & LP, Needs Met & Page Quality Ratings Explanation 


Needs Met: The result block shows three 
popular Target locations in Jacksonville, 
with information that is especially helpful 
for users who want to visit the store. 


The LP is the Target website. 


Needs Met: The result is very satisfying 
for users that want to go to the website. 


Page Quality: Target is a popular 
shopping website in the U.S. with high 
E-E-A-T and is trusted by users for online 
purchases. High to Hight is an 
appropriate rating. 


The LP is the “store locator” page on the 
Target website. 


Needs Met: The LP is more specific than 
the query, but many or most users would 
be interested in this page. 


Page Quality: Target is the expert about 
where its stores are located. This page is 
the most authoritative page on this topic. 


The LP is the “electronics” page on the 
Target website. 


Needs Met: The LP is more specific than 
the query, but many or most users would 
be interested in this page. 


Page Quality: Target is a popular 
shopping website in the U.S. with high 
E-E-A-T and is trusted by users for online 
purchases. High to Hight is an 
appropriate rating. 
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Query and User Intent 


Query: [target] 


User Location: 
Jacksonville, Florida 


User Intent: Go to 
target.com or find a 
nearby Target store. 


Result Block & LP, Needs Met & Page Quality Ratings 


Target 
Corporation 


Retail company 


Stock price: TGT (NYSE) 
$79.84 -1.67 (-2.05%) 


Apr 29, 2:42 PM EDT - Disclaimer 
CEO: Brian C. Cornell 


Founder: George Dayton 


Profiles 


Of 0 B 


YouTube Facebook Linkedin Google+ 


v More about Target Corporation 


| No Page Quality rating 
FM required 


Women's Jegging Black - Cherokee : 
Target 
m.target.com/.../women...jegging-black-c... 
Mobile-friendly - Jan 4, 2015 - Find product 
information, ratings and reviews for a Women's 
Jegging Black - Cherokee. 


jm | 


FailsM FullyM Lowest Highest 
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Explanation 


Needs Met: While this result block has 
some helpful content, most people 
searching for Target are not looking for 
the stock price or the names of the CEO 
and Founder. Slightly Meets 

to Slightly Meets+ is an appropriate 
rating. 


The LP on the Target website is for a 
specific type of women’s pants. 


Needs Met: The LP is much more specific 
than the query and would be helpful for 
very few users. 


Page Quality: Target is a popular 
shopping website in the U.S. with high 
E-E-A-T and is trusted by users for online 
purchases. High to Hight is an 
appropriate rating. 
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Query and User Intent 


Query: [chicken recipes] 


User Intent: Users 
probably want to make a 
chicken dish and are 
looking for some recipes 
to choose from. Users 
probably expect and 
want a list of recipes. 


Chicken Recipes: Food Network : 
Recipes and Cooking : Food ... 
www.foodnetwork.com/.../chicken.html 
Results 1 - 10 of 5500 - Find chicken recipes, 
videos, and ideas from Food Network. 


FullyM Lowest Medium 


Chicken Parmesan Recipe : Tyler 
Florence : Food Network 
www. foodnetwork.com/.../ty... 
zaa Kok Ie Rating: 5 - 427 reviews - 1 
hr 5 mins 


| Mobile-friendly - Get Tyler Florence's 


classic Chicken Parmesan recipe from 
Food 911. 


Lowest Medium 


Fried Chicken Recipes - Allrecipes.com 
m.allrecipes.com/recipes/.../chicken/fried-... 
Mobile-friendly - See more than 80 delicious 
authentic fried chicken recipes with photos and 
tips. ... Staff Picks. Honey Fried Chicken ... 


FullyM Lowest Medium 


Chickenrecipes 

www.chickenrecipes.com/ 

chickenrecipes.com has been informing visitors 
about topics such as Chicken Breast Recipes, Easy 
Chicken Dishes ... 


I a | | 


FailsM FullyM Lowest Medium 
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= 


Highest 


= 


Highest 


Result Block & LP, Needs Met & Page Quality Ratings Explanation 


This LP has many chicken recipes (with 
reviews) on a popular recipe website. 


Needs Met: This LP fits the query. It would 
be very helpful for most users. 


Page Quality: Food Network is a popular 
website with high E-E-A-T and a good 
reputation for providing high quality recipes 
and cooking information. High+ to 
Highest is an appropriate rating. 


This LP has a single recipe for chicken 
parmesan. 


Needs Met: This is a popular type of 
chicken recipe on a popular website, but 
the LP is more specific than the query. 
Some or few users would find this page 
helpful. 


Page Quality: Food Network is a popular 
website with high E-E-A-T and a good 
reputation for providing high quality recipes 
and cooking information. High+ to 
Highest is an appropriate rating. 


This LP has over 25 recipes for fried 
chicken, a popular chicken dish. 


Needs Met: Even though there are over 25 
different recipes on the LP, they are all for 
the same basic dish. Therefore, this LP is 
also more specific than the query. Some 
or few users would find this page helpful. 


Page Quality: Allrecipes.com is a popular 
website with high E-E-A-T and a good 
reputation for providing high quality recipes 
with lots of reviews. High+ to Highest is 
an appropriate rating. 


This is a fake search page for chicken 
recipes. 


Needs Met: It should be rated FailsM as it 
would be helpful for no users. 


Page Quality: Ads should never disguise 
themselves as the MC of the page. Pages 
with Ads that are designed to look like MC 
do not have a beneficial purpose. Since 
this page deceives users, it should receive 
the Lowest rating. 
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Query and User Intent 


Query: [smithsonian air 
and space museum 
store] 


User Location: 
Washington, DC 


User Intent: Go to the 
store website or find 
information, such as 
opening hours, about the 
museum’s store in 
Washington, DC. 


Result Block & LP, Needs Met & Page Quality Ratings Explanation 


Museum Store | National Air and 
Space Museum 

Smithsonian Institution > airandspace > ... 

Bring home a memento of your Museum visit or 
find the perfect gift for someone special. The 
Museum store offers a .. 


l | | | 


Fully Lowest Medium 


= 


Highest 


Screen onthe Green) 1 (6) 
A National Mall 
Smithsonian Castle (E) 7 

Siitgenian Natio 
Air and Space Muse 
nian Metro{) 


Smithsonian National Air 


and Space Museum 
4.6 kk kkk 493 reviews 
National Museum - Southwest Washington 


U ° © 


CALL DIRECTIONS WEBSITE 


Museum documenting the history of aviation & 
space exploration, with displays of vehicles & 
more. 


9 600 Independence Ave SW, 
Washington, DC 20560 


© Open today 10:00 am - 5:30 pm v 


Jore about Smithsonian National Air and Space Muse. 


, No Page Quality rating 
FM required 


Volunteer | National Air and 
Space Museum 

Smithsonian Institution » airandspace > v... 
Mobile-friendly - Support the Museum by 
volunteering your time. Work as a _.. 
Smithsonian Air and Space Museum. 


FullyM Lowest Medium 


= 


FailsM Highest 
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This LP is the official website for the 
museum store. 


Needs Met: The result is very satisfying for 
users that want to go to the website. 


Page Quality: This is the website of a 
popular museum store with a good 
reputation and high E-E-A-T. The museum 
is the expert on information about the 
store, including hours, phone numbers, 
membership information, etc. High+ to 
Highest is an appropriate rating. 


Needs Met: While the result is about the 
museum, the hours, address, and 
directions are the same as the museum 
store. This information is helpful for users 
searching for the store. 


This LP has information about how to 
volunteer at the museum. 


Needs Met: The LP is more specific than 
the query and would be helpful for few 
users. 


Page Quality: This is the volunteer page 
for a popular museum with a good 
reputation and high E-E-A-T. The museum 
is the expert on their own volunteer 
opportunities. High+ to Highest is an 
appropriate rating. 
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Query and User Intent | Result Block & LP, Needs Met & Page Quality Ratings Explanation 


Smithsonian Museum of + 
i : : Natural History Gift Shops 
Query: [smithsonian air 4.4 kkk 14 reviews - $88 


Gift Shop 
and space museum 
store] ù ° © 


CALL DIRECTIONS WEBSITE 


User Location: Series of kid-triendiy shops within the museum 

Washington, DC — ee Needs Met: This is not the museum 
G00 Consttuton Ave Nw. Washo, requested in the query so the result Fails 
User Intent: Go to the DC 20004 to Meet the user intent. 

store website or find C “Shentotey 1008-530 pm x 

information, such as 

opening hours, about the ore about Smithsonian Museum of Natural History G 

museum’s store in 


Washington, DC. 


, No Page Quality rating 
FM required 
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18.0 Needs Met Rating and Freshness 


Some queries demand very recent or “fresh” information. Users may be looking for “breaking news,” such as an important 
event or natural disaster happening right now. Here are different types of queries demanding current/recent results. 


Type of Query Example Queries Explanation 


Assume users need the information right away. 


“Breaking news” queries | [tornado], [tsunami] Imagine someone who needs immediate weather information because 
a big storm is coming. Information about last year's weather would 
not be helpful. 


Assume users are looking for the most recent or current information 
about the event. 
Recurring event queries, 
such as elections, sports 
events, TV shows, 
conferences, etc. 


[olympics], [american idol], 
[redsox schedule], [tax forms], 
[elections] 


For example, if the Olympics are happening right now, users 
searching [olympics] want information about the current Olympics, not 
results from years ago. If the next Olympics are a few months away, 
users are probably interested in the 

upcoming Olympics. 


[population of paris], [amount of 
u.s. debt], [airfare from ny to sfo], 
[next federal holiday] 


Produictquernes [iphone], [toyota camry], [windows | Assume users are looking for information about the most recent 
q operating system] model/version for these product queries. 


Current information 
queries 


Assume users are looking for the most current information, such as 
prices or airfare. 


When a query demands recent content, only pages with current, recent, or updated content should get high Needs Met 
ratings. For these queries, pages about past events, old product models and prices, outdated information, etc. are not 
helpful. They should be considered “stale” and given low Needs Met ratings. In some cases, stale results are useless 
and should be rated FailsM. 


For some queries, there may be “newsy” or recent information user intent, as well as more “timeless” information user 
intent. Users issuing queries for celebrities or politicians may be interested in biographical information, or users may be 
looking for the latest news or gossip. 


Freshness is generally less of a concern for Page Quality rating. “Stale” pages can have high Page Quality ratings. For 
example, some highly reputable news websites maintain “archival” content. However, unmaintained/abandoned “old” 
websites or unmaintained and inaccurate/misleading content is a reason for a low Page Quality rating. 


Note: The date the page was created may be different from when the content was last updated or modified. When 
content is updated, the page will sometimes show the date of the update, not the date the page was created. Some 
websites always show the current date, regardless of when the content was last updated. If you are curious about the 
content of a page, see here to try the “Wayback Machine” on the Internet Archive. Not all pages are available, but this 
tool may help you understand how some pages were created and how their content has changed over time. 
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Query and User Intent 


Query: [nasa mars 
landing] 


User Intent: On the date 
this example was 
written, NASA had just 
landed the “Curiosity” 
rover on Mars. Most 
users are probably 
looking for breaking 
news and pictures about 
the event. 


Query: [los angeles 
traffic] 


User Location: Los 
Angeles, California 


User Intent: Users are 
looking for current 
information about traffic 
conditions in Los 
Angeles. Even pages 
about traffic conditions 
the day before would be 
considered stale for the 


query. 


Query: [nobel peace 
prize winner] 


User Intent: Find the 
name of the most recent 
winner of this prize. This 
is a Know Simple query. 
At the time this example 
was originally written, 
2011 was the most 
recent year that the 
Nobel Peace Prize was 
awarded. 


Curiosity Rover Lands Safely on Mars - 
NYTimes.com 
mobile.nytimes.com/.../curiosity-rover-la... 
Mobile-friendly - Aug 6, 2012 - One of the first test 
images from NASA's Mars Curiosity rover that 
helped signal that everything was ... 


FailsM 


NASA's Curiosity Rover Sets Off for 
Mars Mission - NYTimes.com 
www.nytimes.com/.../nasas-curiosity-rov... 
Nov 26, 2011 - NASA Launches Sophisticated 
Rover on Journey to Mars. NASA/Reuters. A 
rendering of the Mars ... 


Lowest 


Your commute: Live L.A. traffic 
conditions | L.A. NOW | Los Angeles ... 
latimesblogs.latimes.com/.../2012/.../your... 
Mobile-friendly - Your commute: Live L.A. traffic 
conditions. August 7, 2012 | 4:30 pm. Los Angeles . 
Traffic Report by Sigalert.com. 


Your commute: Live L.A. traffic 
conditions | L.A. NOW | Los Angeles ... 
latimesblogs.latimes.com/.../2012/.../your... 
Mobile-friendly - Your commute: Live L.A. traffic 
conditions. August 6, 2012 | 6:30 am. Los Angeles . 
Traffic Report by Sigalert.com. 


FailsM Lowest 


Nobel Peace Prize Awarded to Three 
Activist Women 
www.nytimes.com/2011/.../nobel-peace-... 
Published: October 7, 2011 ... Among 3 Women 
Awarded Nobel Peace Prize, a Nod to the Arab 
Spring (October 8, 2011) ... 


FailsM Fully 


BBC NEWS | Europe | Nobel for anti- 
poverty pioneers 
news.bbc.co.uk/2/hi/europe/6047020.stm 

Oct 13, 2006 - Muhammad Yunus of Bangladesh 
and the Grameen Bank have been jointly awarded 
the 2006 Nobel ... 
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Medium 


Medium 


Medium 


Medium 


Medium 


p=] 


= 


Highest 


Highest 


— | 


Highest 


p 


Highest 


p=] 


= 


Highest 


Highest 


Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


This LP has an article about the 
“Curiosity” rover landing, 
published on the same day as the 
landing, which is the day this 
example was written. 


This LP has an article about the 
launching of the “Curiosity” rover, 
published nine months before its 
landing on Mars. The content on 
this page would be considered 
stale for the query. 


This LP shows traffic conditions in 
Los Angeles on August 7, 2012, 
which is the day this example was 
written. 


This LP shows traffic conditions in 
Los Angeles on August 6, 2012. 
Itis stale and useless for the 
query even though it is only one 
day old. 


This LP on the New York Times 
website has information about the 
most recent winner of this prize, 
based on the date when this 
example was written. 


This LP on the BBC website has 
information about the 2006 winner 
of this prize. This page is stale 
for the query. 
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Query and User Intent | Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


Arthritis: MedlinePlus 
m.medlineplus.gov/arthritis.html 
If you feel pain and stiffness in your body or have 
trouble moving around, you might have arthritis. Medline Plus is an authoritative 
Query: [arthritis] Most kinds of arthritis ... website that regularly updates its 
content. 
User Intent: Users are 
looking for information 
about this disease. Fails 
Because there is 
ongoing research done | Arthritis Drug Approved for Expanded 
on treatments and cures | Use - New York Times 
for most diseases, users www.nytimes.com/2006/05/.../20drug.ht... 
are probably looking for | May 20, 2006 - Federal regulators approved wider This 2006 article would be 
authoritative, up-to-date | use of Johnson & Johnson's drug Remicade but considered stale for this medical 
information on arthritis. also said that a... GUB 


pa | 


FailsM Medium Highest 


Franklin D. Roosevelt - Biography - U.S. 
President - Biography.com 
www.biography.com/.../franklin-d-roose... 
Query: [when was Born on January 30, 1882, in Hyde Park, New York, 
franklin d roosevelt born] Franklin D. Roosevelt was stricken with polio in This LP has the information users 
1921. He became the 32nd U.S. president in 1933, are looking for. 


User Intent: Users want and was the only president to be elected four times. 


to find this well 


understood fact. 
| —| 


Lowest Medium 


We couldn't find a stale page for this query. 


19.0 Misspelled and Mistyped Queries and Results 
19.1 Misspelled and Mistyped Queries 


You will notice that some queries are misspelled or mistyped. Here are some examples of queries that are obviously 
misspelled: 


Misspelled Query Query Interpretation 
[federal expres], English (US) The only reasonable query interpretation is the company named Federal Express. 


[new england patroits], English US) The only reasonable interpretation is the NFL football team. 
[byonce knowles], English (US) The only reasonable interpretation is the famous singer/actress, Beyonce Knowles. 


Some misspelled or mistyped queries are more difficult to interpret. Use your judgment and do query research. 


For obviously misspelled or mistyped queries, you should base your rating on user intent, not necessarily on exactly how 
the query has been spelled or typed by the user. 


For queries that are not obviously misspelled or mistyped, you should respect the query as written, and assume users 
are looking for results for the query as it is spelled. 
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19.2 Name Queries 


Consider the query [john stuart], English (US). There is a very famous Jon Stewart, the comedian and former host of a 
popular U.S. television show. However, we should not assume that the query [john stuart] has been misspelled. There 
are many people named John Stuart. We will respect the query as written and assume the user is looking for someone 


named “John Stuart.” 


Query and Possible 
User Intents 


Query: [micheal jordan], 
English (US) 


User Intent: There are 
many people named 
“Micheal Jordan.” 


Helpful results for more 
prominent people named 
Micheal Jordan should 
receive higher ratings 
than helpful results for 
less prominent people 
named Micheal Jordan. 


Because Michael Jordan 
is such a famous 
athlete/celebrity, we will 
consider Michael Jordan 
to be a minor 
interpretation. 


Micheal Jordan profiles | LinkedIn 
https://www.linkedin.com/.../Micheal/Jor... 
Mobile-friendly - View the profiles of professionals 
named Micheal Jordan on LinkedIn. There are 25 
professionals named Micheal ... 


f= 


FullyM Medium 


Micheal Jordan | LinkedIn 
www.linkedin.com/.../micheal-jordan/.../... 
Atlanta, Georgia - Facilities Services Professional 
Office Supporter at Fort Street U.M. Church; 
Administrater/Facilitator at Fort Street United 
Methodist Church; Supervisor ... 


L 
— i 
FailsM Medium 


NBA.com: Michael Jordan Career Stats 
www.nba.com/history/.../jordan_stats.htm... 

Full Name: Michael Jeffrey Jordan Born: 2/17/63 in 
Brooklyn, N.Y. High School: Laney (Wilmington, 
N.C.) College: North ... 


Lowest Medium 


20.0 Non-Fully Meets Results for URL Queries 


Highest 


Highest 


Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


The page lists all 
members of the LinkedIn 
site named “Micheal 
Jordan.” Linkedin is a 
very popular social 
networking website for 
professionals in the U.S. 
This result would be very 
helpful for many or most 
users trying to find 
information about 
someone named 
“Micheal Jordan.” 


LinkedIn page for 
Micheal Jordan, a 
facilities services 
professional in Atlanta, 
Georgia. 


Michael Jordan’s player 
profile page on the NBA 
basketball website. 


Raters sometimes ask the question, “For a well-formed working URL query, are the only acceptable Needs Met ratings for 
a result either Fully Meets or Fails to Meet?” The answer is no. There can be other helpful results for URL queries. 


Some users issue URL queries to find information about a website, such as reviews or recent news. We recommended 


this to you as one method of reputation research in the PQ guidelines. Real users do this too. Results that give reviews 
and reputation information can be very helpful for a URL query. 


However, websites that offer usage statistics about a website are not usually helpful results for URL queries. Most users 
aren't interested in this kind of information. 
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Query and User Intent 


Query: 
[potterybarn.com] 


User Intent: Pottery 
Barn is a home 
furnishing store. Users 
may want to go to the 
website, do research, or 
find more information on 
the company. 


Query: 
[ratemyprofessors.com] 


User Intent: Go to 
www.ratemyprofessors.c 
om, a website where 
students can rate their 
college professors. 


Pottery Barn 
Pottery Barn > home 


Mobile-friendly - Pottery Barn's expertly crafted 


collections offer a widerange of stylish indoor and 


outdoor furniture, accessories, decor ... 


Up to 75% off Pottery Barn Coupons, Promo 
Coupons.com > ...» Home & Garden 


*& kk k& © Rating: 81% - 37 votes 


Mobile-friendly - 5 verified Pottery Barn coupons and 
promo codes as of May 25. Popular now: Save Up to 


75% Off Sale & Clearance . 


1—.| 
FailsM FullyM 


Pottery Barn Reviews - Consumer Reviews of 
Potterybarn.com | SiteJabber 
www. sitejabber.com > reviews > Www.po.. 


Mobile-friendly - 73 reviews for Pottery Barn, 4.2 stars: "I 


love the stuff here. ... SiteJabber Online Business 
Reviews and Complaints 


Rate My Professors - Review Teachers 
and Professors, School ... 
www.ratemyprofessors.com/ 

Mobile-friendly - Rate My Professors is the best 
college professor reviews and ratings source based 
on student feedback. Over 1.4... 


FailsM 


RateMyProfessors.com Rankings 
Released | Leslie Reece Schichtel 
www.huffingtonpost.com/leslie-reece.../c... 
Mobile-friendly - Aug 31, 2012 - The rankings offer 
good news to both penny-pinching students and 
America at large. Less expensive ... 


Towson's RateMyProfessor gets ready 
to test the market - Baltimore ... 
www.bizjournals.com/.../story8.html?... 

Apr 17, 2006 - Two Towson entrepreneurs bought 
RateMyProfessor.com last fall for seven figures. 
They're already ... 


I rool 
FailsM FullyM 
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Medium 


= 


Medium 


pa 


p | 


Medium 


Medium 


Medium 


= 


Highest 


Highest 


| 


Highest 


Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


This LP is the target of the query. 


Online coupon codes are very 
popular in the U.S. and this LP 
(from a well-known coupon site) 
displays codes for online 
purchases on the Pottery Barn 
website. Users may be interested 
in coupon codes and this would 
be very helpful for those shopping 
online at potterybarn.com, 
especially frequent shoppers. 


This LP (from a well-known 
review site) has reviews of the 
potterybarn.com website. Users 
may be interested in reading 
reviews when considering 
whether to make purchases at 
potterybarn.com, especially new 
shoppers. 


This LP is the target of the query. 


This LP is a Huffington Post 
article dated August 31, 2012 
about ratemyprofessors.com. 
The article was newly published 
when it was added to the 
guidelines. Some users might be 
interested in this article. 


This LP is an article dated April 
17, 2006 about the 
ratemyprofessors.com website. 
Few or no users would be 
interested in this outdated 
information. 
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21.0 Product Queries: Importance of Browsing and Researching 


Some product queries, such as [ipad reviews], have a clear information-seeking (Know) intent. Other product queries, 
such as [buy ipad], have a clear purchase (Do) intent. And some product queries, such as [ipad store.apple.com], have a 
clear navigation (Website) intent. However, most product queries don’t obviously specify one type of intent. 


Keep in mind that many users enjoy browsing and visually exploring products online, similar to window shopping in real 
life. Give high Needs Met ratings to results that allow users to research, browse, and decide what to purchase. 


Users may not always plan to buy products online that they are browsing and researching, for example, cars or major 
appliances. Even though the ultimate goal may be to purchase a product, many other activities may take place first: 
researching the product (reviews, technical specifications), understanding the options that are available (brands, models, 
pricing), viewing and considering various options (browsing), etc. 


Important: Page Quality ratings for product results need extra care and attention. 


Query and User Intent 


Query: [high definition 
tv] 


User Intent: This query 
doesn’t specify a size or 
brand of TV. Users are 
probably doing research 
before making a 
purchase. 


Query: [girl toddler jeans 
size 3T] 


User Intent: The query 
is very specific. The 
user wants to browse 
and find an item to buy. 


Query: [cufflinks] 


User Intent: Most users 
probably want to find 
different cufflink styles 
and options to browse or 
purchase. 


HDTV Reviews | HDTVs Review | 
PCMag.com 

www.pcmag.com/reviews/HDTV 

PCMag provides up-to-date coverage and product 
reviews of HDTVs. 


FailsM Lowest Medium 


Toddler Girl Jeans at babyGap | Gap - 
Free Shipping on $50 

www.gap.com > Home > Toddler Clothing 
Mobile-friendly - Gap toddler girl jeans collection 
includes a variety of popular styles. Pick from 
toddler girls jeans in skinny, wide leg and ... 


FailsM FullyM Lowest Medium 


Cuff Links for Men & More Men's 
Jewelry | Nordstrom 
m.shop.nordstrom.com/c/mens-cuff-links 
Mobile-friendly - Free shipping on cuff links & 
men's jewelry at Nordstrom.com. Shop cuff links, 
tie clips, bracelets & necklaces. Totally ... 


ps -n fj—— | | -m 


FailsM FullyM Lowest Medium Highest 


Copyright 2023 


Result Block and LP, Needs Met Rating, Page Quality Rating Explanation 


This LP has comprehensive 
information about different types 
of high definition TVs (plasma, 
LCD, LED), as well as reviews 
about many specific models. The 
website is known for having 
helpful reviews and information 
about electronics. 


This is a helpful page for 
browsing and shopping for toddler 
girls jeans. There are many 
options for jeans in size 3T. This 
is a high quality store known for 
selling jeans. 


This LP has a large collection of 
cufflinks from a very reputable 
retailer. There are many options 
and styles. This is a good page 
for browsing. There are helpful 
features at the top to choose 
styles, price ranges, etc. 
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Query and User Intent Result Block and LP, Needs Met Rating Explanation 


Query: [twist of fate ring pandora] YW i> This result has a variety of 
images of the ring, showing 
User Intent: The user is looking g ge ~ different angles and materials. 
for a specific ring sold by the 4 i > Knowing what the ring looks like 
Pandora brand and likely wants to A if is very helpful to users who are 
research the ring, see what it : ia researching the product and 
looks like, find options to browse browsing options that are 
or purchase, etc. available. 


ps ; | 
FailsM FullyM 


22.0 Rating Visit-in-Person Intent Queries 


When there is a user location for a visit-in-person intent query and a location has not been specified in the query itself, 
such as [chinese restaurants] with a user location of Boston, MA, results in or near the user location are the most helpful. 


How close is “near”? The type of business and/or entity should be taken into consideration when deciding if the distance 
of the visit-in-person result is too far. For example, most people are not willing to travel very far for a gas station, coffee 
shop, supermarket, etc. Those are types of businesses that most users expect to find nearby. 


However, users might be willing to travel a little farther for certain kinds of visit-in-person results: doctors’ offices, libraries, 
specific types of restaurants, public facilities like swimming pools, hiking trails in open spaces, etc. Sometimes users 
may accept results that are even farther away, such as a very specialized medical clinic. 


In other words, when we say users are looking for results “nearby,” the word “nearby” can mean different distances for 
different queries. As always, please use your judgment. 
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22.1 Examples Where User Location Does (and Does Not) Matter 


As previously mentioned, the user location may not always change our understanding of the query and user intent. Below 


is one example where user location plays an important role, and two examples where user location does not matter. 


Query and User Intent 


Query: [minimum wage 
rate] 


User Location: Wichita, 
Kansas 


User Intent: Find out 
what the current 
minimum wage is. 
Different states in the 
U.S. have different 
minimum wage rates. 
Users might be 
interested in the 
minimum wage in their 
state and how it 
compares to other 
places. 


Denver.) = 
United States kardad © 
)LORADO = —— 7 


KANSAS MISSOURI 


Map data ©2014 Google, INEG 


7.25 USD per hour (Jan 1, 
2014) 


Kansas, Minimum wage 


Destinations and points of interest 


' 
SM 


Pay and Wage Requirements - Kansas 
Department of Labor 
https://www.dol.ks.gov/Laws/FAQwages.... 

Pay and Wage Requirements. Minimum Wage. 
What is the minimum wage in Kansas? The Kansas 
minimum wage is ... 


Minimum Wage Laws in the States - 
Wage and Hour Division (WHD ... 
www.dol.gov > WHD 

Jan 1, 2015 - Interactive map of the United States 
comparing the Federal and state minimum wage 
amounts. 


Minimum wage - California Department 
of Industrial Relations 
www.dir.ca.gov/dise/faq_minimumwage.... 

After your claim is completed and filed with a local 
office of the Division of Labor Standards 
Enforcement (DLSE), it will ... 
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Result Block and LP, Needs Met Rating Explanation 


This SCRB displays the complete and 
correct answer prominently in the result 
block, in a way that is easy for users on 
mobile devices to read. 


Note: Assume that the result shows 
current information for users at the time 
this example was written. If you were 
rating this task, you would need to verify 
that the minimum wage rate in this SCRB 
is accurate for the state of Kansas. 


This is the official Kansas state website 
page on minimum wage for Kansas. 


This LP has the minimum wage rates for 
all states. It is on the official U.S. 
Department of Labor website, which has 
high E-E-A-T on the topic, but it takes 
some work to find the information specific 
to Kansas. 


This is the official California state website 
page on minimum wage for California. 
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Query and User Intent 


Query: [pictures of 
kittens] 


User Intent: Users are 
looking for pictures of 
kittens. This is a 
non-visit-in-person intent 
query. There is no 
obvious user intent to 
find pictures of kittens in 
Pittsburgh. The user 
location plays no role in 
the Needs Met rating. 


Query: [google jobs] 


User Location: Miami, 
Florida 


User Intent: Users are 
looking for job 
opportunities at Google. 


pittsburgh pets - craigslist - Pittsburgh 
- Craigslist 
pittsburgh.craigslist.org/search/pet 
Mobile-friendly - pittsburgh pets - craigslist. ... 
pets. search titles only; has image; posted today. 
reset search. — —. listthumbgallerymap. 


ae PKWY 
Googleplex 
| Charleston Rd 
Googleplex 
4.4 ek Kw 1,527 reviews 


Corporate Campus 


0 Amphitheatre Parkway, Mountain View, CA 94043 


(650) 253-0000 
+ Open today - 9:00 am — 5:00 pm 


& Y © 


Call Directions Website 


More about Googleplex 


Google Careers 
https:/www.google.com/about/careers/ 
Mobile-friendly - About Google: Careers. Google 
Jobs ... Take a ride on the Google self-guided 
tour. Stop by our offices around the globe ... 
Search all jobs - Locations - Teams and Roles 
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Result Block and LP, Needs Met Rating Explanation 


The LP is a page full of kitten pictures. 
The result block has a nice assortment of 
pictures as well that would highly satisfy 
users. 


This is a local listing of pets needing 
homes in the Pittsburgh area. There are 
no pictures of any pets directly on this 
page and few pictures on the individual 
listings. 


The user is looking for a job at Google and 
there is no information in this result block 
that will help the user explore job 
opportunities—it Fails to Meet the user 
need. 


This is Google’s official page on careers 
and jobs at Google, where users can 
search for currently open opportunities. 
This page is exactly what the user is 
looking for—it Fully Meets the user need. 
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23.0 Rating English Language Results in Non-English Locales 


The following rating guidance is for raters in non-English locales. You may stop reading this section if your task 
language is English, such as English (US), English (IN), English (NG), etc. 


Your Needs Met ratings should reflect how helpful the result is for users in your locale. When the query is in the language 
of your locale, assume that users want results in that language. We know that you can read English (you are reading this 
document!), but you should only give high Needs Met ratings to English results if users in your locale would expect or 
want them for a particular query. Unless requested by the query, English results should be considered useless if most 
users in the locale can't read them. 


However, rating can be more difficult when the query includes English names, words, etc., or when it's unclear whether 
English results would be satisfying for a particular query. Please use your judgment and knowledge of your locale to 
determine the appropriate rating. 


Important: Please keep in mind that every locale will have unique considerations regarding the number and variety of 
languages (such as official languages, regional languages, local dialects, etc.), writing systems, and keyboard input 
languages commonly in use. While these guidelines may not include examples for your locale, it is important that you 
represent users in your task location and culture in order to interpret the query and rate results. When in doubt, please 
assume that users would prefer results in the task language unless the query clearly indicates otherwise. 
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23.1 Examples of English (and Non-English) Results in Non-English Locales 


The section includes some examples using Hindi (IN) and Korean (KR) as the locales. In both cases, we cannot assume 
that users in these locales—i.e., Hindi-speaking users in India, or Korean-speaking users in Korea—are able to read 
English. Unless most users in the locale would be satisfied by English results for the query, we will consider them 
unhelpful or even useless (FailsM). 


Query and User Intent 


Query: Rron (fà ar] 
Locale: Hindi (IN) 


User Intent: The user wants to 
find information 
about wen wf aaf, an Indian artist. 


Query: [H= SHO] 
Locale: Korean (KR) 


User Intent: H=} 2yfO} is Barack 
Obama in Korean, so the user 
wants to find information about 
Barack Obama. 


wart Ufa sat - Afra 
https://hi.m.wikipedia.org > wiki > IT... 

AE Hare - TAT TiS TAT (28e - 2806) TNT È 
meaa Rarer ft) Segre ars meea AT... 
staa otra - Pp - tere cer - er pA 


FailsM 


Raja Ravi Varma - Wikipedia, the free 
encyclopedia 

https://en.m.wikipedia.org > wiki > Raja_... 

ago Fad - Raja Ravi Varma Coil Thampuran 
(Malayalam: @o820 @ailaidéam; 29 April 1848 — 2 
October 1906) was an Indian ... 

Laxmi Vilas Palace, Vadodara - Damayanti 


H= QHO - 97/42}, Pe] 272] IMAA 
ko.m.wikipedia.org/wiki/H3}_Q8}0} 
Mobile-friendly - H8 421 Qu}0f 241 CHES (SOI: 
Barack Hussein Obama, Jr., =3}04: Het Suto}, 
19614 8% 4Y ~ , 0/3 5ł2}0] ... 


FailsM 


Barack Obama - Wikipedia 
en.m.wikipedia.org/wiki/Barack_Obama 
Mobile-friendly - Barack Hussein Obama |I is 
an American politician who served as the 44th 
President of the United States from 2009 to 
2017. He is the first African American to ... 
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Result Block and LP, Needs Met Rating Explanation 


This Hindi language Wikipedia 
page on arant ¢fa aaf is very 
helpful for Hindi (IN) users. 


This English Wikipedia page 
about tail tfa Gat has similar 
content to the Hindi Wikipedia 
page. Although there are some 
helpful images on the page, few 
Hindi users would be able to read 
this page. 


This Korean language Wikipedia 
page on Barack Obama is very 
helpful for Korean (KR) users. 


This English Wikipedia page 
about Barack Obama has similar 
content to the Korean Wikipedia 
page. Although there are some 
helpful images on the page, very 
few Korean users would be able 
to read this page. 
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Here are two examples where the query includes proper nouns typed in Latin script, such as famous people, places, titles 
of books or films, etc. For these queries, users would prefer to see results in the language of their locale. 


Query and User Intent 


Query: [ronaldo] 


Locale: Hindi (IN) 


User Intent: The user wants to 
find information about Cristiano 
Ronaldo, a famous footballer. 


Query: [titanic 1997] 
Locale: Korean (KR) 


User Intent: The user wants to 
purchase a DVD or find 
information about the movie 
“Titanic,” released in 1997. 


fafecaret terest - Afra 
https://hi.m.wikipedia.org > wiki > #... 


Bara aan). aAA, (1985-02-05) 5 rat 
1985 ... 


Cristiano Ronaldo - Wikipedia, the free 
encyclopedia 
https://en.wikipedia.org > wiki > Cristian... 


Aaga Heald - Cristiano Ronaldo dos Santos 
Aveiro, GOIH (born 5 February 1985), known as 
Cristiano Ronaldo (Portuguese ... 

Gemma Atkinson - Messi-Ronaldo rivalry 


EFO|E+} —- Daum Sst 
m.movie.daum.net/m/detail/main.daum?... 

SHS AA - 19124 HAYY 27H HRE Sol 
At SCH EY Het SAS EOE H S'7} EACH S| 
Oho AAO] LAE... 


FailsM 


Titanic - IMDb 
m.imdb.com/title/tt0120338/ 

ek kW Rating: 7.7/10 - 640,553 votes 
Mobile-friendly - Titanic -- This made-for-TV 
movie is the tale of the famous - Titanic -- Jack 
discusses his view of the world with the first ... 


FailsM 
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Result Block and LP, Needs Met Rating Explanation 


Although the query was typed in 
Latin script, most Hindi users 
would want to see information 
about this person in Hindi. This 
Hindi language Wikipedia page 
on Cristiano Ronaldo is very 
helpful for Hindi IN) users. 


This English Wikipedia page 
about Cristiano Ronaldo has 
similar content to the Hindi 
Wikipedia page. Although there 
are some helpful images on the 
page, few Hindi users would be 
able to read this page. 


Although the query was typed in 
Latin script, most Korean users 
would expect to see Korean 
language reviews or Korean 
stores for purchasing the movie. 
This Korean result on a Korean 
website is very helpful. 


This English language landing 
page has helpful content for users 
who can read English. Although 
there are some helpful images on 
the page, very few Korean users 
would be able to read this page. 
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Here are two additional examples where the query was typed in Latin script (including some English words), yet users 
would still prefer to see results in the language of their locale. 


Query and User Intent 


Query: [mahila cricket sri lanka 
ka match] 


Locale: Hindi (IN) 


User Intent: The user wants to 
find information about a women’s 
cricket match against Sri Lanka 
on February 17, 2016. 


Query: [samsung tablet] 
Locale: Korean (KR) 


User Intent: The user wants to 
purchase, find information, or go 
to the Galaxy tablet page on the 
Samsung website. 


khabar.ndtv.com > fie 


6 hours ago -... Aa et seier Wz arana ite, tt 
aad A Mta À oft 2-0 Ht arata aga. 


FailsM 


Indian Women's Team Defeats Sri Lanka to 
Seal ODI Series - Cricket News - Sports 
sports.ndtv.com > cricket > 255592-india... 


Mobile-friendly - 6 hours ago - Indian women's 
cricket team defeated Sri Lanka by six wickets to 
take unbeatable 2-0 lead in... 


Atal Weta 
www.samsung.con/.../tablet/.../SHW-M3... 
Gute! 2| Sl - SAMSUNG $C}=,SKT,SHW- 
M380SFK3SC, Samsung GALAXY Tab 10.1 
ZHeA AJEHTO.1 


Am - SHA - D421 - 0/0)2| 


Galaxy Tab: Android Tablet - Portable & 
Light Tablet] Samsung 
www.samsung.com/us/mobile/galaxy-tab/ 
Mobile-friendly - Meet the Samsung Galaxy Tab 
family including Galaxy Tab 10.1, 8.9 and 7.0 Plus. 
Sort by size or carrier and find the ... 

Galaxy Tab - WiFi Tablets 


FailsM 
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Result Block and LP, Needs Met Rating Explanation 


feet cy A sieer We AIM site, aq aes BT 
att 


Although the query includes some 
English words, it is a Hindi query 
typed in Latin script. Most Hindi 
users would expect to see Hindi 
results, like this helpful article. 


This English language landing 
page has helpful content for users 
who can read English, but few 
Hindi (IN) users would be able to 
read this page. 


Although the query was typed in 
Latin script and includes some 
English words, most Korean 
users would expect to see the 
Korean page on the Samsung 
website. This Korean result is 
very helpful. 


This English language page on 
the website of Samsung, the 
company that makes this tablet, 
has helpful content. Although 
there are some helpful images on 
the page, very few Korean users 
would be able to read this page. 
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On the other hand, there may be queries where the needs of most users would actually be satisfied by English language 
results, even if the query itself is typed in the task language. 


For example, for queries about global businesses and organizations, users may expect or want to visit the English 
language version of the business/organization's official website in some locales. Similarly, for queries seeking technical 
information such as manufacturer part numbers, product specs, scientific or chemical formulas, etc., the answer to the 
query may be typically expressed in the English language in some locales. 


For these queries, users may expect or want to see English results in order to satisfy their need. Please use your 
judgment and knowledge of your locale to determine the appropriate rating. 


Query and User Intent 


Query: [elds dage 
Locale: Hindi (IN) 
User Intent: eg daaisc is 


"Harvard website" in Hindi. Users 
want to go to the official website 
of Harvard University, which is a 
well-known research university in 
Cambridge, Massachusetts. 


Query: [JAZ 7}Be|AE AA 
IOS 25H71] 


Locale: Korean (KR) 


User Intent: This Korean query 
can be translated as "recovering 
IOS on Cisco Catalyst switches". 
The user wants to find out how to 
restore the IOS operating system 
on a Cisco Catalyst switch, which 
is a specific brand/model of 
computer networking device. 


Harvard University 
www.harvard.edu 


Harvard University is devoted to excellence in teaching, 
learning, and research, and to developing leaders in many 
disciplines who make a difference globally. Harvard 
University is made up of 11 principal academic ... 
Admissions & Aid - Schools : Faculty : Harvard College 


i} i 
FailsM FullyM 


Cisco Catalyst 43| (2900, 3500, 3750) IOS Sst 
7|: WOH #23 
m.blog.naver.com » 220189570678 


2016. 6. 20. - Cisco IOS Software, C3560 Software (C3560- 
IPSERVICESK9-M), Version 12.2(53)SE, ... Technical 
Support: http://www.cisco.com/techsupport 


FailsM 
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Result Block and LP, Needs Met Rating Explanation 


This is the official website of 
Harvard University. Even though 
the query was typed in Hindi, this 
English website is clearly what 
users are looking for. Users 
would be fully satisfied by this 
result. 


This article comes from a blog 
that contains lots of technical 
information and troubleshooting 
tips about IT operations and 
programming. The landing page 
provides step-by-step instructions 
in Korean, along with the exact 
commands in English that users 
would need to type into a console 
in order to recover the operating 
system for the queried device. 
This tutorial would be very helpful 
and easy to follow for Korean 
users who are familiar with this 
type of networking device. 
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Finally, in some locales, English is one of the official languages or a commonly spoken language. In these locales, 
English websites are easy to use and could be helpful, depending on the query. 


For example, the Singapore government recognizes four official languages: English, Malay, Chinese, and Tamil, but 
English is the first and most dominant language in Singapore. 


Query and User Intent Result Block and LP, Needs Met Rating Explanation 


Barack Obama - Wikipedia 
en.m.wikipedia.org/wiki/Barack_Obama 
Mobile-friendly - Barack Hussein Obama II is 

2 ; an American politician who served as the 44th This Wikipedia page in English 
ARE R President of ho United States from 2009 to about Obama would be very 
User Intent: Find information 2017. He is the first African American to ... helpful to users in Singapore. 
about Barack Obama. 


Query: [barack obama] 


24.0 Rating Dictionary and Encyclopedia Results for Different Queries 


When assigning Needs Met ratings for dictionary and encyclopedia results, careful attention must be paid to the user 
intent. Like all results, the helpfulness of dictionary and encyclopedia results depend on the query and user intent. 


Dictionary and encyclopedia results may be topically relevant for many searches, but often these results are not helpful for 
common words that most people in your rating locale already understand. Reserve high Needs Met ratings for dictionary 
and encyclopedia results when the user intent for the query is likely "what is it" or "what does it mean" and the result is 
helpful for users seeking that type of information. 


Important: Think very carefully about the helpfulness of dictionary and encyclopedia results for ordinary words and 
common items. If few users would benefit from a dictionary or encyclopedia result for a common word, a Slightly Meets 
rating may be appropriate. If very few or no users would benefit, then Fails to Meet is appropriate. 


Query and User Intent Result Block and LP, Needs Met Rating Explanation 


@) Merriam-Webster > dictionary > obs 


Obsequious | Definition of Obsequious by ee 
; , , This is a vocabulary word often 
Query: [obsequious] Merriam-Webster studied in secondary school and 
found on college entrance exams. 
Many users may be looking for a 
definition and would find this 
dictionary result helpful. 


User Intent: Users are probably Obsequious definition is - marked by or exhibiting a 
looking for the definition of this fawning attentiveness. How to use obsequious in a 
vocabulary word. sentence. Follow Along With the ... 


FailsM 
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Query and User Intent 


Query: [rainbow] 


User Intent: This is a broad 
query. Users could be looking for 
many things, including images of 
rainbows, brands or local 
stores/organizations with 
"rainbow" in their name, etc. 


Query: [cafeteria] 
User Location: Eugene, Oregon 


User Intent: Most 
English-speaking users in the US 
are familiar with cafeterias. Users 
are most likely looking for a place 
to eat. 


Query: [history of ATM machines] 


User Intent: Find information 
about the history of ATM 
machines (e.g., the inventor, 
when it was invented, its 
evolution, etc.). 


(=) Merriam-Webster > dictionary > rai... 


Rainbow | Definition of Rainbow by Merriam- 


Webster 


Rainbow definition is - an arc or circle that exhibits in 


concentric bands the colors of the spectrum and that is 
formed opposite the sun by the refraction and ... 


W Wikipedia > wiki» Cafeteria 


Cafeteria - Wikipedia 


A cafeteria, sometimes called a 
canteen outside the U.S., is a type of 
food service location in which there is 
little or no waiting staff table service, 
whether a restaurant or within an 
institution such as a large office 


building or school; a school dining location is also referred 


to as a dining hall or lunchroom (in ... 


History Other names College 


W Wikipedia > wiki » Automated_telle... 


cafeteria 


Automated teller machine - Wikipedia 


Jump to History +- On September 2, 1969, Chemical Bank 
installed the first ATM in the U.S. at its branch in Rockville 


Centre, New York. The first ATMs were designed to 


dispense a fixed amount of cash when a user inserted a 


specially coded card. 


Location Financial networks 
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Global use 


Hardwar 


Result Block and LP, Needs Met Rating Explanation 


Most users probably already 
know what a rainbow is and 
would not be looking for a 
definition. Even if the user needs 
help understanding the word, 
images or a simple description 
would be more helpful than this 
dictionary result. 


Most users are looking for a place 
to eat rather than general 
information about cafeterias. 
Even though this is an in-depth 
encyclopedia article that matches 
the topic of the query, it would be 
helpful for few users. 


The user wants to learn more 
information about the history of 
ATM machines. This 
encyclopedia article includes an 
extensive section on ATM 
machine history. This result 
would be very helpful for users 
looking to learn more about the 
history of ATM machines. 
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Query and User Intent 


Query: [atm near me] 


User Location: Livingston, 
Wyoming 


User Intent: Find ATMs near the 
user location. 


Query: [atm] 
User Location: El Paso, Texas 


User Intent: Most users are 


probably looking for ATMs nearby. 


Fewer users would be looking for 
general information about ATMs 
(history, how they work, etc.). 


Result Block and LP, Needs Met Rating Explanation 


W Wikipedia > wiki» Automated_telle... 


Automated teller machine - Wikipedia 


Jump to History +- On September 2, 1969, Chemical Bank ; ; 
installed the first ATM in the U.S. at its branch in Rockville The user intent is to find a nearby 
Centre, New York. The first ATMs were designed to ATM machine. This encyclopedia 


i fi í h ; d article fails to meet the need of 
dispense a fixed amount of cash when a user inserted a thé user because thas no. 


specially coded card. information about nearby ATM 
locations. 
Location Financial networks Global use Hardwarı 


W Wikipedia > wiki» Automated_telle... 


Automated teller machine - Wikipedia 


Jump to History +- On September 2, 1969, Chemical Bank 
installed the first ATM in the U.S. at its branch in Rockville This encyclopedia article about 
Centre, New York. The first ATMs were designed to ATMs is a minor interpretation of 


dispense a fixed amount of cash when a user inserted a the query. This result would be 
iali coded card helpful for fewer users; most 
es ee ee users would be looking for 


locations of ATMs nearby. 
Location Financial networks Global use Hardwar 
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Appendix: Using the Evaluation Platform 


25.0 Overview 


The Evaluation Platform is the system you will use to acquire and rate tasks. You can access the Evaluation Platform by 
going to this link: http:/Awww.raterhub.com/evaluation/rater 


As a Search Quality Rater, you will work on many different types of rating tasks. This section provides guidance on how 
to use certain features of the Evaluation Platform that you will encounter frequently in Page Quality (PQ) and Needs Met 
(NM) rating tasks. Some of these concepts may apply to other types of rating tasks as well. 


Important: Many tasks that you receive will display project-specific instructions on the task page. You should always 
follow these project-specific instructions when they are provided. 


26.0 Acquiring Tasks 


Here is a simplified screenshot of the task acquisition page on the Evaluation Platform. Please note that changes are 
sometimes made to the appearance of this page, so it may not look exactly as it appears below. 


Evaluation Platform rater@gmail.com Personalized task history Recent tasks Sign out $$ 


Desktop Tasks 


Acquire if available | Headphones or speakers required 


Acquire if available 


A few notes about how to acquire tasks from this page: 
e rater@gmail.com: Make sure you are logged in to the account you used to sign up for the rating program. 
e Acquire if available: Click this button to acquire a new task. 


e Headphones or speakers required: This label indicates that the task requires headphones or speakers (for 
example, to listen to an audio file). 


Please note you may see other types of labels next to the button on the task acquisition page. Sometimes there 
will be only one button, while other times you may see more than one button displayed to distinguish tasks with 
different requirements. When there is more than one displayed, you may choose the one you want to acquire. 


27.0 Rating Tasks Using the Rating Interface 


Here is a general outline of the steps you should follow upon acquiring a task: 
1. Read the instructions: Pay close attention to any project-specific instructions and examples if provided. 
2. Fill out the task: Assign ratings to result blocks, provide comments, answer questions, etc. (as applicable). 


3. Check your work: Verify that you entered your ratings as intended. Additionally, make sure that you provided a 
response to all required questions. 


4. Submit the task: Click the "Submit" button at the end of the page to submit the task and continue rating. 
Alternatively, click "Submit and Stop Rating" to submit the task and return to the Evaluation Platform homepage. 
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Additionally, here are a few general things to keep in mind when rating tasks: 


e Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make 
sure that all result blocks have loaded. Please note that result blocks may be slow to load and/or that the task 
page may need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few 
times, please release the task by following the instructions in Section 28.0 below. 


e Sometimes a side will not have any result blocks: If one side displays the message “This side did not generate 
any results,” please do not release the task. 


e Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will 
lose them and they will have to be entered again. 


28.0 Releasing Tasks 


Most raters have difficulty rating some tasks now and then. Some queries are about highly technical topics (e.g., 
computer science or physics) or involve very specialized areas of interest (e.g., gaming or torrents). 


You will report problems and release tasks using the “Report a Problem / Release this Task” button in the rating interface. 
Here is what the “Report a Problem / Release this Task” button looks like: 


Report a Problem / Release this Task 


Clicking the red button displays a menu with several reporting and release options: 


Report a Problem / Release this Task 


Select One 
D) | lack expertise for this task. 
D Uncomfortable rating this adult content. 
D) Uncomfortable rating this upsetting-offensive content. 


~ The instructions or task are too confusing or unclear (please 
describe). 


D) The task is in the wrong language. 
D The average estimated time is too low. 


, | do not understand the query or user intent, even after doing 
~ some research. 


_) | do not meet the upfront requirements for this task. 


~ Cannot access content behind paywall (please paste the URL 
into the comment box). 


D) There is a technical problem with this task (please describe). 


D Other (please describe). 


Comment 


Release this task? 
© Yes - Allow me to select a new task. 


J No - Send my feedback but let me continue to work on this task. 


Will a screenshot help us understand the issue? 


J Yes - Please take a screenshot for me. 


@No - A screenshot is not needed. 


Submit Report 


Please note that changes are sometimes made to this menu, so it may not look exactly as it appears above. 
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In general, you should choose whichever option best describes your reason for releasing the task. If you are unsure, 
please select the closest option, or contact your employer/company for further guidance. 


Please leave a comment explaining the release when it makes sense to provide additional information, for example, when 
you are releasing for a “technical problem.” 


Here are some examples for the "lack expertise” and “technical problem” release options. 


Lack expertise: These queries require specialized knowledge and may be difficult for some raters. 


[cortisol secretion shows a blank pattern of secretion] Scientific query 


[minecraft gun and vehicles 1.1 mod installer] Gaming query 
[python pandas index like matrix] Technical query 


Technical problem: These result blocks have question marks and other characters where text should be. 


[uD sSORE°021-68861966 18918321661 www. - ieee 
11 U UZ b0 U $ °021-68861966 


STETE TWN tianxiay.com - AiE tE 
www.dearkay.com/qd_www.tianxiay.com.html 


hit 


Same as R1 


2??? 2? 222222? 2222? 22227 windows 7 ultimate ??? 


Note: Based on the number and/or type of tasks that you release, you may be asked to provide additional details about 
the reason for some of the releases. 


Copyright 2023 171 


29.0 Understanding the User Location on the Task Page 


The location of the user should be inferred from the map provided. The map can help clarify where the user was 
physically located when the query was issued. 


On the map, the user location can be precise (blue dot), approximate (blue rectangle or circle), or missing altogether if the 
user location is unknown. Here are some examples of what you might see at the top of the task: 


A precise user location 
is represented by a blue 


dot( ° ). 


San Francisco 


The user can be located 
anywhere within the 
blue rectangle on the 
map. 


Sometimes you will see 
a shaded blue circle 
around the blue dot. The 
blue dot is the user’s 
most likely location, but 
it is possible they are 
located anywhere within 
the larger shaded blue 
region. 


The map includes a 
response pin, which is a 
marker for a 
visit-in-person result 


block in the task. If 
San Francisco svi 
Qe ae 7 ny aiie there are three 

} | visit-in-person result 
blocks in the task, there 
should be three 
response pins on the 
map. 


30.0 Reporting Duplicate Results in Tasks 


In some rating tasks, you may be asked to determine whether any results are Duplicates (dupes). This section provides 
guidance and examples of how to report duplicate results In Needs Met (NM) rating tasks. 


30.1 Pre-Identified Duplicates 


Some duplicate results are automatically detected and will be pre-identified for you. They will be annotated by the text 
“Same as...” right below the slider and above the result block. 


This is what these pre-identified dupes look like: 


Left Side 


Right Side 


L2 - Same as R2. - select dupes 


THE 10 BEST Charlotte Hotels with a Pool of 2019 (with Prices .. 

https://www.tripadvisor.com » North Carolina (NC) >» Charlotte » Charlotte Hotels + 
te Hotels w P ’ 6846 t 

6 hotels 


Needs Met Rating Page Quality Rating 


t t | ł | F } j ł | 
N/A FailsM SM HM FulyM N/A Lowest Low Medium High Highest 


Comment 


Porn: No Foreign Language: No Did Not Load: No 


R2 - Same as L2. - select dupes 


THE 10 BEST Charlotte Hotels with a Pool of 2019 (with Prices .. 

https://www. tripadvisor.com > North Carolina (NC)> Charlotte » Charlotte Hotels + 

E l} e Hotels wit 3 N P I T li 6846 t € revie 1 
phote 4 oricas for 116 hotels 


Needs Met Rating Page Quality Rating 
t | | t | | | | 
N/A FailsM SM HM FullyM N/A Lowest Low Medium High Highest 


Porn: No Foreign Language: No Did Not Load: No Comment 


Please note: You cannot unselect dupes that have been automatically detected and pre-identified. 
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30.2 Rater-Ildentified Duplicates 


You are also asked to help identify duplicate results that have not been automatically detected. Please mark two results 
as dupes if they have essentially the same content on the main landing page AND you would not want a search engine to 
return both results for the query. 


Please note that in Needs Met rating and other query-based tasks, dupe identification is query-dependent. 


e Specific queries: For queries where the user is looking for a specific piece of content (such as queries looking for 
song lyrics, queries looking for a specific article, etc.), obtaining that piece of content from different sites could 
be helpful for users to verify the information, so they should not be rated as dupes. 


e Broad queries: If the query is broad, then returning the same piece of content is not what the user is looking for, 
and hence those results should be flagged as dupes. Results may be considered dupes even if they have minor 
content differences on the page (such as different ads, images, or related links). 


Please identify dupes both within the same side and across sides. Even for cross-side results, you should still ask 
yourself the question, “Would users want to see both results if they were returned by the same search engine?” 


Note: If two result blocks have very different types of content or very different appearances, do not mark them as 
duplicates even if they have the same landing page URL. For example, these two blocks should not be marked as 
duplicates of each other. 


Sunnyvale CaltrainE} 
RR 


Dishdash Dishdash 

4.4 kkk kk 522 revien www.dishdash.com/ 

SSA E EEES Mobile-friendly - Catering. Please visit our 
Dishdash family of restaurants. ddgrill.png 
dnd.png - yelp.png - FB100px.png - 
instagram.png 

Dinner - Lunch - Our Story - Order 


> Open today 


N 7 © 


Call Directions Website 
Menu: locu.com 


More about Dishdash 
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30.3 Reporting Duplicate Results 


When you notice that the results in two or more result blocks are duplicates, please click on the Select Dupe link at the 
top of one of the results. The current result (the result you clicked on) will be highlighted in red at the top of the block. 


R5 - finish selecting dupes 


HOUSTON ZOO - NATURALLY WILD - Home 

www. houstonzoo org/ 

Features information and pictures on the different animals as well as zoo hours 
facility map, classes and programs, and 

45 kkk ke 290 review 


You can then check results that are duplicates of the current result, and the checked results will be highlighted in yellow. 
You must select the “Dupe of...” button. 


2Q- Dupe of R5: No 


HOUSTON ZOO - NATURALLY WILD - Home 

www.houstonzoo org/ 

Features information and pictures on the different animals as well as zoo hours, 
facility map, classes and programs, and 

45 kkkKSD review 


The duplicate results that you checked will be annotated by "Dupe of..." text at the top of the block. After selecting all 
dupes, click on the finish selecting dupes link to return to the normal rating mode. The link’s name will change back to 
Select Dupes, and you will be able to report other sets of dupes, if there are any. If you change your mind, you can 
always uncheck a result. 


Dupes 
QUERY: [choosing and installing a motorcycle battery] 
URL 1: http://www.caimag.com/wordpress/2010/03/06/motorcycle-battery-how-to-choose-install 


URL 2: http://www.articlesbase.com/motorcycles-articles/choosing-and-installing-a-motorcycle-battery-47798.html 
Reason: Both of these results display the same article (which also appears on many other pages on the web). The only 


real difference between the landing pages are the Ads displayed around the article. The query is broad enough that users 
would not benefit by search engines returning more than one of these results. 


Dupes 
QUERY: [jason castro] 
URL 1: http://www.myspace.com/jasoncastromusic 


URL 2: http://www.myspace.com/jasoncastromusic?MyToken=503599bf-01cf-4427-bdf4-d63920c107f9 
Reason: These two results have the same landing page, even though the URLs are different. Users would not benefit by 
search engines returning both results. 


Not Dupes 

QUERY: [material girl lyrics] 

URL 1: http://www.lyricsfreak.com/m/madonna/material+girl 20086925.html 

URL 2: http://www.lyrics007.com/Madonna%20Lyrics/Material%20Girl%20Lyrics.html 

Reason: Even though both pages display the lyrics to the song “Material Girl,” users would probably want to have the 
option to visit both pages so that they could verify the accuracy of the lyrics. Users could benefit by search engines 
returning more than one page with the lyrics to the song. 
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31.0 Simplified Needs Met Tasks 


Some Needs Met rating tasks may not require ratings for all blocks. In these blocks, the Needs Met and Page Quality 
rating scales will be greyed out and annotated with the text "No Rating Required". 


The greyed-out results (sometimes referred to as "contextual results") are shown to help you understand the query and 
inform your ratings. You can click on them when you feel like they will be helpful in determining your ratings. 


Here is an example: 


L1 - select dupes R1 - Same as L2 - select dupes 


tevision otat i; WWAY News - Twitter 
vision sta paivan 


79 3K tweets + 15K photos/videos - 40 8K followers. 
Check out the latest Tweets from WWAY News 


(@wway) 
WWAY is the ABC/CBS/CW-affiliated television station for the Cape Fear region of Nort) Carolina in the United States 
that is boensed to Wilmington Wiupedia WWAY Sports 
wner Morris Multimedia WWAY Weather 
to: Live. Local. interactive. CBS Has a New Home on WWAY (on DTZ) Dare to Defy (on OT3) 
od: 1984 Neecs Met Rabng Page Qualty Rating 
k tte Jt tt tt tt 
v NA FaisM SM MM HM FullyM NIA Lwst Low Med High Hghst 
More about WWAY Porn: No Foreign Language: No Did Not Load: No Comment 
Neode Met Rating R2 - Same as L3. - select dupes 
L aama aam saaa mann auma amma amma aam | 
“ ARE Ss Ue Anm WECT News (@wectnews) | Twitter 
Por: No Foreign Language: No Did Not Load: No Comment Twitter > wecinews 
LE -Guneet AL- eslit ages WWAY News @WWAY ... Christopher White, the victim 
in last night's shooting at Jungle Rapids, has passed 
WWAY News - Twitter —_ 
Twitter » wway Neecs Met Rang Page Qualty Rating 
Me ttt ttt tt tt ttt 
79.3K tweets + 16K photoshideos + 40.0K folowers NA FoluM SM MM HM FulyM NIA Lwst Low Med High Hghst 
Check out the latest Tweets from WWAY News Gomes 
(@wway) Porn No Foreign Language: No Did Not Load: No J 
WWAY Sports R3 -Some as L4. 
WWAY Weather #WWAY - Twitter Search 
Twitter » search » qeaWWAY 
Noods Met Rating Page Qualty Rabi 
y ft td E HH A descnpbon for this result is not avażabie because of 
NA Fave sm MM HMO FulyM NA Lwst Low Mea High Hghst this site's robots.txt 
Pom: No Foreign Language: No Did Not Load: No Comment cam mere 
No Rating Required 
L3 - Same as R2 - select dupes Neecs Met Rating Page Quatty Rating 
P t ta fun fecha) y $ | $ 
WECT News (@wectnews) | Twitter WA FaleM SM MM HM FulyM NA Lwst Low Med High Hohst 
Twitter » wectnews Porn: No Foreign Language: No Did Not Load: No Comment 


WWAY News @WWAY .... Christopher White, the victim 
in last night's shooting at Jungle Rapids, has passed 


R4 -Same as L5. 


away 
WECT Sports (@wectsports) | Twitter 
Needs Met Rating Page Qualty Rating Twitter » wectsports 
E +—+—+—_+—_}-_+_+-_+{ E Hoa al 
NIA Fae SM MM HM FutyMm NA Lwst Low Mes High Hghst WWAY Sports @WWAYSports - Tim Hower- 
Pom No ‘Foreign Language: No! [Did Noi Load: No Convecrd SNVarstyVertied account @ StarNewsVarsity Joe 


Catenacci @JoeCats 19. 
LA - Same as R3. No Rating Required 
Page Quaity Rating 
#WWAY - Twitter Search j t tł 
Twitter » search » q=#WWAY WA FalsM SM MM HM FulyM N/A Lust Low Med High Hhst 
Pom: No Foreign Language: No Did Not Load: No Comment 


} 


A description for this result is not avatable because of 
this site's robots tet RS 


No Rating Required Sports - WWAY 
Needs Met Ratng Page Qualty Rating WWAY TV3 > sports 
+ 1+ t l t t i 
NIA w MM HM Fum NA Lwst Low Keseiowski races way into Ird round of NASCAR's 
playofts - Martin Truex Jr... WWAY Sports on Twitter. 


Pom:No Foreign Lamy No Did Not Load: No 
gn Language My Tweets 


LS - Same as R4 No Rating Required 
Needs Mot Rating Page Quality Rating 


WECT Sports (@wectsports) | Twitter t t O a | { y $ } $ 
Twitter » woctaports NA FabM SM MM HM FubyM NA Lat Low Med High Hghst 


Pom: No Foreign Language: No Did Not Load: No Comment 


WWAY Sports @WWAYSports Tim Hower- 
SNVarsayVerified account @ StarNewsVarsity Joe 
Catenace @JoeCats19 


No Rating Required 
Nowds Met Rating Page Quality Rating 


E t H j(—þ—j] t t 


NIA F HM Fum NA Lwst Low 
Pom: No Foreign Language: No Did Not Load: No 
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